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EXECUTIVE  SUMMARY 


The  goals  of  the  Susceptibility  Model  Assessment  and  Range  Test  (SMART)  Project 
are  to  develop  an  efficient  process  for  the  verification  and  validation  (V&V)  and 
configuration  management  (C/M)  of  aircraft  susceptibility  models  and  to  facilitate 
their  quick  and  cost  effective  accreditation  by  applying  this  new  approach  to  five 
models  frequently  used  in  survivability  analyses  supporting  acquisition  decisions. 
The  credibility  assessment  process  thus  defined  generates  various  reports  that 
provide  a  model  user  with  enough  information  on  existing  verification,  validation 
(V&V),  and  configuration  management  (C/M)  data  to  permit  accreditation  with  a 
minimum  of  additional  effort.  To  make  these  reports  most  beneficial  to  accreditation 
proponents,  the  SMART  Project  undertook  a  study  of  accreditation  requirements. 

The  study  involved  a  review  of  existing  instructions  and  directives,  as  well  as 
interviews  with  personnel  engaged  in  all  aspects  of  model  development  and  use. 
The  analysis  focused  on  two  aspects  of  typical  accreditation  decisions:  accreditation 
processes  and  information  requirements.  The  discussions  about  comparative 
accreditation  processes  and  requirements  generated  significant  interest  from  a  wide 
group  of  DoD  and  service  officials,  and  so  this  accreditation  process  analysis  is 
reported  separately  from  the  findings  on  accreditation  information  requirements. 

The  accreditation  processes  defined  in  current  and  emerging  service  directives 
generally  require  totally  independent  VV&A  with  outside  experts  or  officials  to 
conduct  it  or  approve  the  results.  Their  purpose  is  to  ensure  uniformity  and  prevent 
bias  in  any  accreditation  decisions.  These  policies  were  found  to  be  excessively 
(and  on  occasion  needlessly)  bureaucratic;  they  involve  numerous  formal  reports 
and  reviews,  generally  have  centralized  the  accreditation  authority,  and  require 
extensive  coordination  to  obtain  meaningful  contributions  from  the  specified  review 
participants.  Despite  the  high  degree  of  process  formality,  the  effectiveness  of  these 
processes  in  yielding  quality  study  results  is  implicitly  dependent  on  the  capabilities 
and  interest  of  the  various  review  bodies  or  officials. 

In  contrast  to  the  stated  policies,  current  accreditation  practices  are  relatively 
streamlined  in  that  the  Verification,  Validation  and  Accreditation  (VV&A)  efforts  are 
merged  into  each  study  process.  The  underlying  VV&A  approach  involves  defining 
the  study  measures  of  effectiveness  (MOEs)  or  measures  of  performance  (MOPs), 
deriving  model  acceptance  criteria,  gathering  V&V  information  to  compare  with  the 
criteria,  and  accrediting  the  model  based  on  that  comparison.  The  procedures  used 
to  implement  this  approach  vary  widely,  and  do  not  appear  to  be  a  major  concern  of 
typical  accreditation  agents.  Existing  practices  are  much  less  formal  than  stated 
policies  would  seem  to  require.  The  emphasis  of  current  accreditation  practices  is 
on  collecting  and  evaluating  sufficient  information  to  determine  model  suitability  for 
the  intended  application,  regardless  of  the  process  by  which  such  information  is 
gathered  and  reviewed. 

Our  analysis  shows  that  the  current  VV&A  approach  appears  to  be  efficient  and 
logical  and  tends  to  mitigate  potential  biases  by  requiring  that  application  specific 
accreditation  criteria  be  defined  before  model  selection.  The  effectiveness  of  this 
approach  depends  explicitly  on  the  competence  and  integrity  of  the  study  analysts. 


The  major  disadvantages  noted  are:  1)  potential  inconsistencies  in  approach 
implementation;  2)  potential  biases  of  the  analysts  coupled  with  resource  limitations 
that  might  cause  incomplete  data  gathering  and/or  improper  comparisons;  and  3)  a 
possible  lack  of  uniformity  and  experience  in  establishing  acceptance  criteria  for 
each  specific  application.  Any  VV&A  policies  that  are  being  developed  should  fulfill 
the  intent  of  the  DoD  guidance  yet  capitalize  on  the  advantageous  features  of  the 
current  practices  while  minimizing  the  disadvantages. 

A  notional  concept  for  model  accreditation  that  fulfills  the  DoD  goals  of  uniformity 
and  objectiveness  and  incorporates  the  efficiency  of  the  current  practices  while 
minimizing  the  variability  was  developed.  This  concept  decentralizes  accreditation 
responsibility,  places  V&V  responsibility  on  the  study  analyst,  and  provides  for 
independent  expert  analysts  to  review  the  study’s  MOEs  and  resultant  model 
acceptance  criteria  where  necessary.  With  expert  concurrence  on  these  two  items, 
the  study  analysts  perform  the  necessary  V&V  to  ensure  that  the  model  satisfies  the 
explicitly  stated  M/S  acceptance  criteria  tailored  to  the  application  at  hand.  Since 
the  primary  burden  of  thought  is  placed  up  front,  the  accreditation  process  avoids 
the  costs  associated  with  pro  forma  execution  of  V&V  procedures  not  tailored  to 
individual  applications. 

The  V&V  reports,  configuration  management  procedures,  and  accreditation  support 
database  produced  by  the  SMART  Project  facilitate  accreditation  and  are  easily 
integrated  into  this  notional  process.  Incremental  application  of  the  SMART  process 
allows  development  of  the  baseline  V&V  products  during  the  course  of  several 
specific  applications  without  excessive  expenditures  on  the  part  of  any  one  user. 

In  summary,  the  notional  process  suggested  in  this  report  coupled  with  the 
incremental  application  of  the  SMART  V&V  process  would  lead  to  the  use  of 
appropriate,  accredited  models  with  minimal  time  and  cost  impacts  on  the  study 
sponsor.  To  put  this  notional  concept  into  practice  the  following  steps  should  be 
implemented: 

1.  DoD  and  service  accreditation  policies  should  focus  on  development  and  use 
of  comprehensive  MOEs/MOPs  and  model  acceptance  criteria.  Requirements 
for  obligatory,  independent  reviews  and  formal  reports  should  be  minimized. 

2.  DoD  and  service  agents,  responsible  for  M/S  policy  development,  should 
charter  one,  or  a  few,  team(s)  of  expert  analysts  to  facilitate  development  and 
review  of  MOEs/MOPs  and  acceptance  criteria  when  requested. 

3.  The  Defense  Modeling  and  Simulation  Office  (DMSO)  should  develop 
guidelines,  tailored  to  each  class  of  applications,  for  defining  MOEs/MOPs. 

4.  DMSO  should  sponsor  a  demonstration  effort  in  which  the  notional 
accreditation  process  is  used  and  independently  evaluated. 

5.  DMSO  should  seek  out  opportunities  to  create  automated  or  expert  systems 
that  will  assist  analysts  in  defining  MOEs/MOPs  and  acceptance  criteria  for 
each  application. 
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1.  BACKGROUND  AND  PURPOSE 


The  goal  of  the  Susceptibility  Model  Assessment  and  Range  Test  (SMART)  Project 
is  to  develop,  document,  establish,  and  transition  a  process  for  assessing  the 
credibility  of  models  and/or  simulations  (M/S)  used  in  the  aircraft  survivability 
discipline.  The  impetus  for  this  goal  is  to  support  acquisition  decisions  (e.g.,  Cost 
and  Operational  Effectiveness  Analyses  [COEAs]  and  Defense  Acquisition  Board 
[DAB]  reviews);  test  planning  and  analysis  during  the  development  test  (DT)  and 
operational  test  (OT)  phases  of  a  program;  mission  planning;  and  many  other 
applications  requiring  analysis  of  the  combat  effectiveness  of  military  aircraft.  The 
M/S  credibility  assessment  process  involves  elements  of  verification,  validation 
(V&V)  and  configuration  management  (C/M).  Implementation  of  this  process  yields 
the  core  information  required  for  accreditation  of  survivability  M/S  used  in  analyses. 
SMART  is  producing  baseline  V&V  and  C/M  information  on  a  range  of  aircraft 
survivability  M/S  and  building  an  accreditation  support  data  base  that  allows  M/S 
users  to  learn  from  and  build  upon  the  work  of  previous  M/S  users.  In  this  way,  M/S 
accreditation  requirements  in  support  of  acquisition  are  more  easily  identified,  and 
accreditation  is  more  easily  (and  cheaply)  performed. 

To  maximize  the  utility  of  SMART  V&V  and  C/M  documentation  to  the  accreditation 
process,  it  became  necessary  to  understand  what  information  is  required  to  support 
accreditation.  Since  a  single  source  of  accreditation  information  requirements  did 
not  exist,  SMART  commissioned  a  study  to  identify  the  requirements  of  the  various 
activities  that  use  and  accredit  M/S  across  the  services.  It  was  intended  that 
comparison  between  these  requirements  and  the  current  SMART  products  provide 
useful  insight  into  what  information  should  be  added  to  make  SMART  products  most 
useful  to  potential  accreditation  proponents. 

Although  the  initial  objective  of  this  study  was  to  determine  the  most  necessary  and 
frequently  used  information  to  accredit  models  and  to  tailor  SMART  products 
accordingly,  the  focus  of  the  study  was  expanded  when  it  became  apparent  that 
emerging  accreditation  policies  differed  markedly  from  current  practices.  These 
differences  had  to  be  understood  to  determine  if  the  current  information 
requirements  would  be  likely  to  change  as  the  new  policies  were  implemented. 
Therefore,  a  more  detailed  analysis  of  the  various  accreditation  processes  was 
undertaken  as  an  adjunct  to  the  information  requirements  analysis.  The  results  of 
this  excursion  are  the  subject  of  the  present  report.  The  companion  volume, 
“Information  Requirements  in  Support  of  Accreditation,”  is  available  as  Report 
number  JTCG/AS-93-SM-20,  Volume  II. 

2.  STUDY  APPROACH 

Information  for  this  study  was  collected  through  personal  interviews  and  document 
reviews.  Individuals  in  the  M/S  community  were  contacted  to  determine  the  key 
DoD  and  service  organizations  and  activities  that  use  survivability  M/S  and  perform 
VV&A  in  conjunction  with  their  use.  A  list  of  these  organizations  was  developed,  and 
points  of  contact  at  each  organization  were  identified  (see  Appendix  A).  An 
interview  guide  was  prepared  to  obtain  consistency  between  interview  results  and  to 
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ensure  that  all  critical  questions  and  issues  were  addressed.  The  interview  guide  is 
contained  in  Appendix  B. 

The  organizations  on  the  interview  list  were  visited  and  those  persons  who  perform 
V&V,  develop  VV&A  policies,  or  approve  or  accredit  M/S  for  use  within  their 
respective  organizations  were  interviewed.  Besides  questioning  the  interviewee 
using  the  prepared  guide,  any  written  material  that  addressed  VV&A  guidelines  and 
established  policy,  or  that  documented  actual  VV&A  efforts,  was  obtained. 
Information  on  other  suggested  points  of  contact  was  also  requested,  thereby 
broadening  the  interview  base  as  widely  as  possible.  Summary  notes  documenting 
the  substance  of  each  interview  were  prepared  and  provided  to  each  interviewee  for 
review  and  comment.  This  practice  ensured  that  interview  statements  were  clearly 
understood  and  that  no  bias  was  introduced  into  the  interview  summaries. 
Summaries  of  the  interviews  are  provided  as  Appendix  C.  They  are  organized  in  the 
order  shown  on  the  interview  list. 

The  service  level  documents  that  contain  the  VV&A  policies  were  then  analyzed  to 
understand  and  compare  official  accreditation  procedures.  The  procedures  were 
reduced  to  process  flow  diagrams  for  analysis.  Local  directives  and  interview 
summaries  were  analyzed  to  develop  descriptive  process  flow  diagrams  of  the 
procedures  actually  followed.  Some  process  flow  diagrams  were  compared  and 
consolidated  wherever  justified  by  substantial  similarities.  The  resultant  procedural 
flow  diagrams  were  briefed  and  discussed  at  meetings  attended  by  personnel  from 
the  parent  activities  of  the  various  interviewees,  and  were  revised  based  on 
comments  received.  Finally,  the  official  and  practical  process  diagrams  were 
compared  and  analyzed  to  determine  similarities  and  differences,  advantages  and 
disadvantages. 

In  comparing  and  contrasting  accreditation  policies  with  actual  M/S  user  practices, 
several  observations  on  the  practicality  and  effectiveness  of  both  policies  and 
practices  emerged.  These  observations  suggested  a  set  of  goals  that  should  be  the 
foundation  of  an  effective  and  efficient  accreditation  process,  and  that  might  form  the 
basis  of  more  practical  policy  directives.  Using  these  goals  as  a  springboard,  a 
notional  accreditation  process  was  formulated  which  integrated  the  most  desirable 
features  of  the  various  official  accreditation  policies  and  actual  practices  into  a 
process  that  leads  from  a  clear  problem  definition,  through  model  selection, 
accreditation,  and  use,  to  a  set  of  supportable  study  conclusions.  The  suggested 
process  ties  in  the  key  information  requirements,  identified  through  the  interviews, 
with  the  goals  for  an  efficient  and  effective  study  process,  to  produce  an  accredited 
model  that  is  suitable  for  use  in  a  given  application. 

This  report  provides  an  in  depth  explanation  of  existing  or  emergent  service  policies, 
actual  accreditation  practices  and  a  suggested  accreditation  process.  It  identifies 
conclusions  derived  from  the  study,  formulates  an  improved  notional  process  that 
incorporates  the  SMART  V&V  approach,  and  presents  recommendations  for 
adopting  and  testing  the  notional  process  with  the  goal  of  formulating  consistent, 
efficient,  and  practical  accreditation  policies  across  the  services. 
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3.  INTERVIEW  FINDINGS 


Very  few  of  the  interviewees  had  experience  with,  or  had  been  involved,  in  a  formal, 
documented  accreditation  of  a  model  or  simulation.  Only  the  Army  Materiel  Systems 
Analysis  Activity  (AMSAA)  and  the  Army  Operational  Test  and  Evaluation  Command 
(OPTEC)  had  conducted  formal  model  accreditations.  Those  interviewees  pointed 
out  the  significant  amount  of  time  and  resources  required  for  a  formal  accreditation. 
Several  other  interviewees  were  involved  in  planning  for  or  conducting  their  first 
accreditation. 

Several  common  perceptions  and  concerns  were  voiced  by  many  of  the 
interviewees.  The  most  significant  concerns  were:  1)  resource  constraints;  2)  lack  of 
common  V&V  techniques  and  approaches;  and  3)  problems  associated  with 
selecting  a  model  for  a  study.  The  major  points  gleaned  from  these  summaries  are 
highlighted  in  the  following  paragraphs.  Detailed  interview  summaries  are  presented 
in  Appendix  C. 

3.1  Resource  Constraints 


Several  interviewees,  most  notably  those  at  the  Center  for  Naval  Analyses  (CNA) 
and  AMSAA,  mentioned  VV&A  cost  as  an  important  consideration.  In  their  view,  the 
value  added  by  collecting  extensive  information  to  support  accreditation  must  be 
weighed  against  the  resources  required  to  complete  the  accreditation  and  the 
importance  of  the  decision.  Concerns  were  also  expressed  about  the  cost  of  formal 
configuration  management  procedures.  They  felt  it  would  be  beneficial  to  know  the 
cost  of  typical  V&V  efforts,  as  well  as  the  cost  of  applying  the  SMART  V&V  process 
to  other  models.  (SMART  is  addressing  this  in  its  FY  94  tasking.) 

Other  interviewees  pointed  out  that  there  is  seldom  enough  time  to  carry  out  a 
formal  model  accreditation  for  a  particular  application  since  the  study  results  are 
often  required  in  the  matter  of  a  few  weeks.  The  organizations  faced  with  this  type 
of  problem  are  the  Army  Deputy  Chief  of  Staff  for  Operations  (DCSOPS),  the  Naval 
Air  Systems  Command,  Warfare  Analysis  Division  (NAVAIR-526),  the  Navy 
Operational  Test  and  Evaluation  Force  (OPTEVFOR),  and  the  Air  Force 
Aeronautical  Systems  Command  (ASC).  Occasionally,  the  lack  of  time  forces 
analysts  to  use  models  that  are  not  really  suited  to  the  study  but  are  either  the  only 
ones  available  or  the  only  ones  with  which  the  study  analysts  are  familiar. 

3.2  V&V  Techniques 

Several  V&V  techniques  were  mentioned  as  being  valuable  in  supporting 
accreditation.  Although  almost  all  interviewees  recognized  the  desirability  of 
performing  extensive  verification  code  checks  and  in  depth  comparisons  between 
model  results  and  real  world  data,  funding  and  time  constraints  frequently  preclude 
such  extensive  V&V.  Instead,  many  model  users  turn  to  other  less  costly  methods 
that  are  also  less  beneficial.  Such  methods  typically  include  reviews  of  past  usage 
and  VV&A  results,  face  validation,  and  some  comparisons  between  models. 
OPTEC  and  AMSAA  have  employed  automated  V&V  tools  (CASE  tools)  with  some 
success,  however,  the  analysts  require  a  good  understanding  of  the  tools  available 
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and  how  they  are  best  applied.  According  to  the  CNA  personnel,  the  most 
appropriate  use  of  CASE  tools  is  to  document  the  software;  current  model  users  are 
generally  hindered  by  poor  documentation. 

Many  organizations,  faced  with  a  lack  of  time  and  resources  to  perform  in  depth 
V&V,  have  relied  on  prior  V&V,  face  validation,  and  benchmarking  as  the  basis  for 
informal  model  accreditation.  Among  those  who  use  these  methods  are  the  Army 
Aviation  and  Troop  Command  (ATCOM),  the  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC),  AMSAA,  OPTEC,  ASC,  and  NAVAIR.  A  common 
view  is  that  past  usage,  coupled  with  reviews  by  subject  matter  experts  (SMEs)  or 
comparisons  between  models,  are  sufficient  to  justify  model  selection  and  use. 
Detailed  comparisons  between  the  model  results  and  test  data  are  made  by 
organizations  engaged  in  test  programs  such  as  AFOTEC  and  OPTEC.  These 
comparisons  are  made  in  parallel  with  the  analysis  being  performed  using  the 
model.  Several  interviewees  specifically  mentioned  the  value  of  understanding  M/S 
assumptions  and  limitations,  and  of  performing  a  sensitivity  analysis  to  build 
confidence  in  model  results. 

A  prime  factor  mentioned  in  a  number  of  interviews  is  the  importance  of  well- 
qualified  analysts.  Air  Force  Studies  and  Analysis  Agency  (AFSAA),  ATCOM,  CNA, 
and  some  divisions  in  ASC  rely  heavily  on  analysts  to  understand  the  models  they 
use  and  to  ensure  that  they  will  produce  reliable  results.  They  feel  that  an  analyst 
who  uses  a  few  models  on  a  regular  basis  knows  model  limitations  well  and  is  best 
suited  to  determine  if  the  model  is  applicable  to  a  particular  study.  This  corporate 
memory  is  perishable,  however,  and,  without  good  M/S  documentation,  can  result  in 
M/S  being  used  just  because  they  have  always  been  used. 

One  other  concern  related  to  V&V  techniques  is  data  validation.  Many  organizations 
addressed  this  issue  in  the  documentation  provided  as  part  of  the  interview.  Both 
OPTEC  and  ASC  personnel  stressed  that  study  validity  was  directly  related  to  using 
valid  data  from  an  approved  source  for  a  study. 

3.3  Model  Selection  Concerns 


One  point  brought  out  by  both  the  OPTEC  and  the  Air  Warfare  Center  (AWC) 
interviewees  seems  obvious,  but  is  so  important  that  it  bears  repeating.  Using  an 
accredited  model  does  not  guarantee  valid  study  results.  Correct  MOPs  or  MOEs 
for  the  study  must  be  chosen  before  the  study  begins.  The  use  of  inappropriate 
MOPs  or  MOEs  can  lead  to  erroneous  results,  even  if  the  model  is  an  accurate 
representation  of  the  system  being  studied.  Appropriate  MOEs  or  MOPs  should  be 
derived  from  the  study  objectives.  This  point  cannot  be  overemphasized, 
considering  the  relationship  between  study  MOEs  and  model  acceptance  criteria,  a 
point  that  will  be  amplified  in  later  paragraphs.  OPTEC  has  a  guide  for  choosing 
their  M/S  acceptance  criteria,  and  AMSAA  has  included  M/S  acceptance  criteria  as  a 
major  item  in  their  accreditation  reviews.  A  sample  set  of  questions  for  defining  M/S 
acceptance  criteria  is  provided  in  Appendix  D. 

Another  issue  mentioned  by  a  large  number  of  interviewees  was  configuration 
management.  Knowledge  of  all  changes  to  a  model  that  make  it  different  from  the 
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one  described  in  the  documentation  is  essential  to  determining  its  suitability  for  a 
particular  application.  Many  also  expressed  concerns  over  the  time  it  takes  to  get 
proposed  and  needed  changes  incorporated,  tested,  and  distributed  to  the  users. 

The  expressed  concerns  about  configuration  management  point  out  the  importance 
of  being  able  to  relate  V&V  information  to  a  particular  version  of  a  model.  This 
concern  is  the  basis  for  the  SMART  Project's  efforts  to  develop  common 
configuration  management  requirements  and  guidelines  that  are  linked  to  the  V&V 
process,  so  that  the  user  has  a  means  of  relating  the  V&V  results  to  all  applicable 
model  versions.  These  FY94  efforts  are  being  undertaken  in  conjunction  with 
government  and  industry  M/S  developers. 

4.  ANALYSIS  OF  ACCREDITATION  POLICIES 

DoD  Directive  5000.59,  entitled  “DoD  Modeling  and  Simulation  (M&S)  Management” 
contains  the  following  direction:  “The  DoD  Components  shall  establish  verification, 
validation,  and  accreditation  (VV&A)  policies  and  procedures  for  M/S  applications 
managed  by  the  DoD  Component.  The  ‘DoD  M/S  Executive  Agent’  for  a  general- 
use  M/S  application  shall  establish  VV&A  procedures  for  that  application.”  In 
response  to  this  direction  the  Army  has  promulgated  Army  Regulation  (AR)  5-1 1 
(“Army  Model  and  Simulation  Management  Program”.)  The  Navy  is  developing  a 
parallel  instruction.  Two  significantly  different  Navy  drafts  were  circulated  for 
comment  during  1993.  The  latest  draft  contains  some  of  the  philosophy  from  the 
previous  drafts  but  is  significantly  different  in  many  respects.  The  Air  Force  has 
established  an  office  within  the  headquarters  staff  (Code  XOM)  and  is  developing  Air 
Force  M/S  management  policies  that  include  VV&A  guidelines.  At  the  time  of  this 
writing,  these  policies  were  also  in  draft  form  and  were  undergoing  review. 

4.1  Army  Accreditation  Policy  Description; 

The  Army  M/S  accreditation  process  is  promulgated  in  AR  5-11.  This  regulation 
states  that  the  defined  process  is  applicable  to  new  model  developments.  However, 
analysis  shows  that  it  could  be  used  equally  well  for  existing  models.  The 
prescribed  process  involves  collection,  review  of  specific  types  of  information  by 
certain  assigned  proponents  in  a  series  of  sequential  steps,  and  reporting.  These 
steps  include  a  V&V  review,  an  accreditation  review  for  a  class  of  applications,  and 
an  accreditation  review  for  a  specific  application.  The  regulation  gives  examples  of 
the  types  of  information  that  should  be  reviewed  at  each  step  in  the  process.  The 
process  and  information  requirements  defined  in  AR  5-11  are  amplified  and 
explained  in  an  Army  pamphlet,  DA  PAM  5-11.  (This  pamphlet  was  available  in  its 
final  draft  form  at  the  time  this  study  report  was  prepared.) 

A  "class  of  applications"  is  defined  as  a  generic  set  of  purposes.  Examples  of 
different  classes  of  applications  are:  research  and  development;  test  and  evaluation; 
education  and  training;  production  and  logistics;  or  analysis  (including  COEAs)  in 
support  of  system  acquisitions.  A  specific  application  is  the  use  of  a  model  in  a 
particular  problem.  An  example  would  be  a  COEA  on  a  proposed  weapon  system. 
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AR  5-11  specifies  the  accreditation  process  and  requirements  for  a  class  of 
applications  and  for  specific  applications.  Accreditation  for  a  specific  application 
presupposes  accreditation  for  the  appropriate  class  of  applications.  A  diagram 
depicting  the  accreditation  process  for  a  class  of  applications  is  shown  in  Figure  1 . 

As  indicated  in  the  figure,  the  V&V  proponent  is  responsible  for  preparing  a  V&V 
plan  for  each  model  that  is  to  be  accredited  for  a  class  of  applications.  This  V&V 
plan  includes  management  information  on  tasks,  schedule,  and  resources  as  well  as 
analysis  information  such  as  scope,  limitations,  constraints,  methodology,  sources  of 
data,  and,  most  importantly,  M/S  acceptance  criteria.  When  V&V  is  completed,  the 
results  are  detailed  in  a  report  that  includes  a  description  of  the  real  world  entity 
being  modeled,  the  intended  purpose  of  the  M/S,  the  design  requirements,  a 
description  of  the  V&V  that  was  performed,  the  extent  to  which  the  M/S  met  the  V&V 
acceptance  criteria,  and  any  recommendations  for  M/S  modifications  to  improve 
acceptability. 


FIGURE  1  ARMY  ACCREDITATION  PROCESS  (CLASS  OF  APPLICATIONS) 


The  accreditation  proponent  is  responsible  for  developing  an  accreditation  plan  in 
conjunction  with  the  V&V  proponent.  This  accreditation  plan  identifies  the  members 
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of  the  accreditation  team,  resources  and  milestones,  documentation  required,  M/S 
acceptance  criteria,  and  proposed  accreditation  methodology. 

When  the  V&V  report  is  completed,  the  accreditation  proponent  is  responsible  for 
conducting  a  complete  review  of  the  development  and  use  of  the  M/S,  its 
documentation,  the  V&V  plans,  V&V  reports,  configuration  control  provisions,  M/S 
assumptions,  previous  usage,  and  previous  user  acceptances.  This  review  is 
documented  in  an  accreditation  report  that  contains  the  background,  description  of 
the  M/S,  acceptability  criteria,  evaluation  of  the  M/S,  and  any  results  and 
recommendations.  This  review,  documented  in  the  accreditation  report,  serves  as 
the  basis  for  the  accreditation  decision,  which  is  itself  subject  to  review  by  the  Army 
Model  and  Simulation  Executive  Council  (AMSEC). 

Besides  being  accredited  for  a  "Class  of  Applications,”  each  model  must  also  be 
accredited  for  each  specific  application.  This  type  of  accreditation  builds  on  the 
previous  accreditation,  particularizing  it  to  the  analytical  problem  for  which  the  model 
will  be  used.  Figure  2  depicts  the  accreditation  process  for  specific  applications. 


Application  Sponsor 


Accreditation  Proponent 
(Class  of  Applications) 


FIGURE  2  ARMY  ACCREDITATION  PROCESS  (SPECIFIC  APPLICATION) 


The  application  sponsor  (the  organization  using  the  model  for  a  specific  application) 
is  the  accreditation  proponent.  The  sponsor  ensures  that  the  model,  as  modified  by 
scenario,  data  inputs,  or  other  changes,  will  provide  results  that  meet  the  essential 
needs  of  the  intended  use.  The  application  sponsor's  review  addresses  data  inputs, 
scenarios,  and  operator-analysts  who  will  use  the  model.  The  results  of  this  specific 
accreditation  are  documented  in  a  brief  accreditation  report  forwarded  to  the 
accreditation  proponent  for  that  class  of  applications. 

If  a  model  not  accredited  for  a  class  of  applications  is  proposed  for  use  in  a  specific 
application,  it  must  undergo  a  more  rigorous  process  to  be  accredited  for  the  specific 
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application.  This  rigorous  process  is  tantamount  to  fulfilling  the  accreditation 
requirements  for  a  class  of  applications  as  well  as  for  the  specific  application. 

The  various  VV&A  responsibilities  are  assigned  to  either  the  V&V  Proponent  or  the 
Accreditation  Proponent.  Assignment  of  V&V  Proponent  or  Accreditation  Proponent 
responsibilities  depends  on  the  stage  of  M/S  development.  During  model 
development,  the  agency  responsible  for  that  development  is  typically  assigned  as 
V&V  Proponent.  For  M/S  that  are  maintained  by  a  single  Army  agency,  that  agency 
is  the  V&V  Proponent.  If  an  M/S  is  under  Army  control  but  has  multiple  users,  the 
agency  chairing  the  M/S  Users  Group  will  be  the  V&V  Proponent.  M/S  users  are 
typically  assigned  Accreditation  Proponent  responsibilities.  Table  1  identifies  the 
V&V  Proponent  and  the  Accreditation  Proponent  for  specific  circumstances. 


TABLE  1  ARMY  VV&A  RESPONSIBILITIES 


V&V  Proponent 

Accreditation  Proponent 

Model  In  Development 

Sponsor 

N/A 

Developed  &  Used  by  a 
Contractor 

Sponsor 

Sponsor 

Used  or  Maintained  by 
Single  Army  Agency 

User 

User 

Multiple  Users 

User's  Group  Head  or 
Predominant  User 

Each  User  for  Each 

Specific  Application 

Other  Service  Control, 

Used  in  Army  or  Joint 

Study 

N/A 

Army  Agency  Responsible 
for  Mission/Function  Area 

Threat  M/S 

Appropriate  Intelligence 
Agency 

Each  User  for  Each 

Specific  Application 

4.2  Army  Accreditation  Policy  Features 

The  Army  process  entails  preparation  of  five  different  VV&A  plans  or  reports  to 
obtain  accreditation  for  a  specific  application.  (This  assumes  that  a  class 
accreditation  is  obtained  first.)  Because  of  the  nature  of  the  V&V  material,  these 
documents  will  most  likely  be  quite  lengthy  and  costly  to  prepare.  In  all  likelihood 
they  will  not  be  read  by  the  accreditation  authority  but  will  only  serve  as  proof  that 
the  model  was  actually  accredited  in  the  event  that  study  results  prove  inaccurate  or 
unacceptable.  One  advantage  of  such  in  depth  reports  is  that  they  document  V&V 
results  for  the  benefit  of  future  model  users  and  reduce  the  likelihood  of  duplicative 
efforts. 

As  indicated  in  Table  1,  Army  VV&A  responsibilities  are  typically  assigned  to  the 
sponsor  or  user.  In  most  cases  the  V&V  Proponent  and  the  Accreditation  Proponent 
will  be  from  the  same  activity  or  command.  Therefore,  accreditation  decisions  are 
made  by  the  agency  or  activity  that  either  commissions  or  conducts  a  study.  This 
feature  allows  the  agency  to  better  coordinate  the  VV&A  plans  and  to  tailor  the 
VV&A  requirements  to  the  needs  of  the  study.  Keeping  the  accreditation  decision 
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within  the  using  activity  minimizes  the  cost  and  time  expenditures  associated  with 
briefing  several  external  activities  for  coordination  and  approval.  Army  policy 
includes  provisions  for  the  review  of  application-specific  accreditation  decisions  by 
the  accreditation  proponent  for  a  class  of  applications,  and  by  the  AMSEC  for 
accreditation  decisions  on  a  class  of  applications.  However,  these  reviews  do  not 
appear  to  cause  any  additional  briefings  or  reports. 

The  advantages  of  allowing  user  accreditation  lie  in  the  efficiencies  that  can  be 
achieved.  These  advantages  are  exemplified  by  AMSAA,  which  has  accredited  at 
least  two  different  models  under  the  guidelines  of  this  regulation.  Since  the 
accreditation  authority  is  the  director  of  AMSAA,  the  V&V  results  were  provided  to 
the  director  in  a  formal  briefing  and  the  annotated  briefing  materials  served  as  both 
the  V&V  and  the  Accreditation  Report.  This  documentation  practice  saved 
considerable  money  and  time  that  otherwise  would  have  been  spent  in  preparing  a 
more  formal  report. 

In  addition  to  spelling  out  the  process  to  be  followed  and  the  responsibilities  of  each 
proponent  in  satisfying  accreditation  requirements,  AR  5-1 1  also  identifies  the  types 
of  information  that  should  be  used  to  justify  model  accreditation.  This  information  is 
amplified  in  the  pamphlet,  DA  PAM  5-1 1 ,  which  also  gives  examples  of  acceptability 
criteria  that  might  be  used  in  judging  model  suitability.  By  identifying  typical 
information  needs  and  giving  examples  of  acceptance  criteria,  the  Army  guidelines 
help  provide  uniformity  in  accreditation  decisions.  Acceptance  criteria  were 
frequently  mentioned  as  one  of  the  key  requirements  to  accredit  a  model  for  an 
application.  The  list  of  sample  acceptance  criteria  provided  in  DA  PAM  5-11  is 
particularly  useful  since  this  is  the  only  document  found  which  provides  such  an 
example.  The  Army  examples  of  acceptance  criteria  are  listed  in  Appendix  E. 

4.3  Navy  Accreditation  Policy  Description 

The  Navy  guidelines  for  M/S  accreditation  are  promulgated  in  a  draft  OPNAVINST 
titled  "Verification,  Validation,  and  Accreditation  (VV&A)  of  Navy  Models  and 
Simulations"  dated  7  October  1993.  This  document  is  supplemented  by  another 
draft  document  titled  “Principles  of  VV&A  of  Navy  Managed  M/S”  dated  1  October 
1993.  These  documents  have  been  circulated  for  comment  and  are  subject  to 
change.  However,  they  do  reflect  the  current  Navy  thinking  regarding  M/S 
accreditation.  The  policies  expressed  in  these  documents  will  be  used  herein  as  an 
indicator  of  probable  Navy  direction  in  this  area. 

Like  the  Army  regulation,  these  documents  focus  on  the  process  for  model 
accreditation.  In  the  proposed  Navy  process,  an  M/S  Proponent  is  responsible  for 
preparing  and/or  collecting  information  on  the  model  and  associated  V&V.  An 
independent  official,  appointed  by  the  Navy  Model  and  Simulation  Management 
Office  (NMSO),  is  responsible  for  certifying  the  acceptability  of  the  V&V  data.  Based 
on  this  certification,  the  senior  management  official  responsible  for  decisions  derived 
from  model  use  accredits  the  model.  The  process  is  depicted  in  Figure  3. 
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FIGURE  3  NAVY  VV&A  PROCESS  FOR  MATURE  M/S 

The  M/S  Proponent  (the  organization  seeking  the  accreditation)  initiates  the  process 
by  assembling  and  submitting  to  NMSO  information  on  design  specifications  and 
requirements,  intended  use(s),  history,  background,  and  prior  V&V.  The  NMSO 
head,  also  referred  to  as  the  Technical  Assistance  Team  (TAT)  Executive,  appoints 
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a  Senior  Technical  Assistance  Director  (STAD)  from  the  TAT  to  guide  the  initial 
review  of  the  material  submitted  by  the  proponent.  This  review  results  in  either  a 
V&V  certification  or  a  recommendation  for  additional  V&V.  If  further  V&V  is  to  be 
performed,  the  STAD  assembles  a  Phase  I  assessment  team  from  a  body  of  M/S 
experts.  The  team  develops  a  V&V  plan  based  on  the  intended  use,  user 
requirements,  model  features,  and  TAT  Executive  direction.  The  plan  includes 
actual  V&V  procedures  to  be  employed  and  recommendations  for  membership  on 
the  Phase  II  review  team. 

The  M/S  Proponent  reviews  the  plan,  especially  the  funding  requirements,  and 
determines  whether  to  proceed  or  postpone  accreditation.  If  the  plan  is  to  be 
executed,  the  proponent  provides  the  necessary  funding.  The  TAT  Executive 
assembles  the  Phase  II  team,  which  implement  the  V&V  plan.  The  Phase  II  team 
forwards  the  results,  along  with  a  recommendation,  to  the  TAT  Executive,  who 
prepares  a  V&V  certification.  This  is  passed  to  the  M/S  Proponent  who  accredits  the 
model.  The  draft  guidance  does  not  address  the  issue  or  possible  impacts  of  a 
decision  to  postpone  accreditation. 

The  draft  Navy  instruction  also  states  that  V&V  should  be  made  an  integral  part  of 
M/S  development,  thus  simplifying  the  VV&A  process  by  providing  crucial 
information  early  in  the  development  cycle.  The  supplemental  “Principles”  document 
describes  how  V&V  should  be  performed  and  documented  in  parallel  with  model 
development  to  produce  an  accreditation  package.  Figure  4  depicts  the  concept  for 
integrating  VV&A  with  M/S  development.  In  this  concept,  the  M/S  Proponent 
ensures  that  the  product  of  each  step  in  the  development  is  checked  with  the 
previous  products  to  ensure  accuracy.  The  results  of  this  comparison  or  check  are 
documented.  This  collection  of  documented  V&V  results  is  reviewed  by  the  TAT 
Executive,  identified  in  Figure  3.  If  the  documentation  is  sufficient,  the  TAT 
Executive  issues  a  V&V  certification,  which  serves  as  the  basis  for  accreditation. 

Just  as  in  the  Army  process,  a  Navy  model  can  be  accredited  for  either  a  specific 
application  or  a  “domain”  of  applications,  which  is  equivalent  to  the  Army’s  “class  of 
applications.”  Unlike  AR  5-11,  domain  accreditation  is  not  a  prerequisite  for 
application-specific  accreditation;  however,  it  will  usually  facilitate  such  an 
accreditation. 

The  supplementary  Navy  document,  “Principles  of  VV&A  of  Navy  Managed  Models,” 
defines  four  different  levels  of  accreditation;  each  level  varying  in  the  depth  of  V&V 
required  for  accreditation.  The  determination  of  accreditation  level  is  dependent  on 
a  trade-off  between  the  level  of  risk  in  using  possibly  inaccurate  results  versus  the 
cost  of  accreditation  at  a  higher  level.  The  requirements  for  each  level  of 
accreditation  assume  the  completion  of  all  elements  specified  for  lower  levels.  The 
M/S  Proponent  recommends  (and  the  TAT  Executive  approves)  the  level  of 
accreditation  required.  The  VV&A  information  requirements  for  each  level  are 
shown  in  Table  2.  The  description  of  each  level  contained  in  the  principles 
document  are  summarized  in  the  following  paragraphs. 

Level  1  accreditation  establishes  and  documents  a  model's  development  and 
enhancement  history,  summarizes  past  applications,  documents  V&V  status,  and 
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defines  its  application  domain  based  on  a  description  of  the  capabilities  prepared  by 
the  model  developer. 

Level  2  accreditation  is  based  on  an  examination  of  M/S  assumptions,  algorithms, 
architecture,  implementation,  development  and  enhancement  history,  past  and 
intended  applications,  and  past  VV&A  efforts.  In  some  cases,  portions  of  the 
software  code  may  be  verified  independently  by  a  review  team,  and  some  test  cases 
run. 

Level  3  accreditation  is  based  on  the  validation  of  M/S  application  results.  It  requires 
correlation  of  M/S  results  with  all  known  data  about  behavior  of  the  subject  modeled. 

Level  4  accreditation  implies  extraordinary  efforts  to  assure  accuracy  and  reliability 
in  M/S  applications.  It  provides  assurance  that  a  model  performs  acceptably  for 
interaction  that  leads  directly  to  the  control  and  movement  of  real  systems  in  real 
time.  This  level  of  accreditation  may  also  be  appropriate  for  other  M/S  whose 
accuracy  is  essential  for  human  safety. 


FIGURE  4  NAVY  VV&A  PROCESS  FOR  MODELS  IN  DEVELOPMENT 

4.4  Navy  Accreditation  Policy  Features 

The  proposed  Navy  instruction  does  not  include  a  requirement,  nor  does  it  discuss  a 
need,  to  define  and  establish  acceptance  criteria  by  which  to  judge  whether  a  model 
is  suitable  for  use  in  a  particular  application.  The  draft  instruction  does  require  a 
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TAT,  led  by  a  STAD,  to  review  and  certify  that  model  V&V  results  are  satisfactory. 
To  do  this  the  TAT  must  study  the  particular  application,  including  the  measures  of 
performance  (MOPs)  or  measures  of  effectiveness  (MOEs),  derive  a  set  of 
acceptability  criteria,  evaluate  whether  a  model  meets  those  criteria,  and  determine 
if  a  model  has  had  sufficient  V&V  to  assure  that  it  can  be  reliably  used  within  the 
envelope  of  the  intended  application.  To  perform  these  functions,  the  TAT  and 
STAD  require  an  understanding  of  the  intended  application  and  any  related  factors 
that  might  affect  development  of  acceptance  criteria.  They  also  need  a  high  degree 
of  M/S  technical  knowledge  to  translate  an  application’s  MOEs/MOPs  into  a 
reasonable  set  of  M/S  acceptance  criteria  and  to  assess  whether  a  model,  its 
operating  limitations,  and  its  V&V  results  meet  those  criteria.  Nevertheless,  the 
critical  relationship  between  study  MOEs/MOPs  and  M/S  acceptance  criteria  is  left 
unexplained  in  the  draft  instruction. 


TABLE  2  INFORMATION  REQUIREMENTS  BY  ACCREDITATION  LEVELS 


Requirement 

Accreditation 

Level 

1 

2 

3 

4 

M/S  history,  intended  use,  V&V  status,  capabilities 

X 

X 

X 

X 

User’s  Guide 

X 

X 

X 

X 

Software  Design  Documents  &  Programmer’s  Manual 

X 

X 

X 

Analyst’s  Manual 

X 

X 

Summary  of  M/S  assumptions,  algorithms,  architecture,  and 

X 

X 

X 

available  input  data 

Verification  plan 

X 

X 

X 

X 

Documented  results  of  verification  tests 

X 

X 

X 

Independent  verification  reviews  and  tests 

X 

X 

Face  validation 

X 

X 

X 

Comparisons  with  real  world  data 

X 

X 

Configuration  management  version  tracking  &  archiving 

X 

X 

X 

X 

Configuration  management  plan 

X 

X 

Tracking  of  Software  Trouble  Reports  and  Change  Requests 

X 

X 

Configuration  management  board  constituted 

X 

Input  data  validity 

X 

X 

Users  and  analysts  trained 

X 

X 

User  groups  established 

X 

User  certification 

X 
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The  proposed  instruction  does  not  define  the  requirements  for  membership  on  the 
TAT  or  for  selection  as  a  STAD.  Membership  on  the  TAT  will  probably  be  selected 
by  the  TAT  Executive,  possibly  assisted  by  various  technical  advisors.  Recognizing 
that  the  TAT  Executive  position  will  usually  be  filled  by  a  commissioned  officer,  and 
that  periodic  rotation  of  these  officers  will  occur,  there  is  no  assurance  that  the  TAT 
Executive  will  always  have  adequate  M/S  expertise  to  select  the  best  personnel  with 
both  M/S  and  subject  matter  knowledge  to  serve  on  the  TAT.  Furthermore, 
considering  the  broad  scope  of  potential  applications,  it  would  seem  nearly 
impossible  to  create  a  standing  team  of  M/S  experts  who  collectively  have 
sufficiently  broad  knowledge  of  potential  applications.  Consequently,  the  quality  of 
decisions  affecting  model  accreditation  will  likely  be  inconsistent. 

Another  result  of  the  lack  of  acceptance  criteria,  or  a  guide  for  developing  them,  is 
the  potential  for  using  a  model  outside  the  envelope  in  which  the  V&V  results  apply. 
Comparisons  between  model  outcomes  and  real  world  results  are  made  under 
specific  environmental  conditions.  If  the  model  outcomes  and  the  real  world  results 
are  closely  related  at  one  or  more  specific  points,  it  is  an  indication  that  the  model 
can  yield  valid,  real  world  predictions  at  other  points  in  the  environment.  However, 
there  is  usually  a  boundary  beyond  which  the  accuracy  of  these  predictions 
becomes  suspect.  TAT  members  will  be  required  to  recognize  these  boundary 
conditions  and  assess  the  extent  to  which  V&V  results  can  be  considered  reliable. 
In  addition,  the  TAT  members  will  also  be  required  to  determine  the  envelope  of  the 
intended  application  and  compare  it  to  the  V&V  boundaries.  This  is  a  tall  technical 
order,  especially  for  a  team  whose  members  are  not  eminently  familiar  with  the 
study  and  its  objectives. 

The  TAT  and  STAD,  lacking  guidance  on  acceptance  criteria,  will  be  motivated  to 
ensure  that  they  will  not  be  criticized  for  recommending  accreditation  without 
sufficient  V&V.  Consequently,  they  will  have  a  tendency  to  be  overly  cautious  and 
require  more  V&V  than  might  be  necessary  if  clear  acceptance  criteria  were 
established.  This  tendency  will  be  unchecked,  since  the  TAT  and  STAD  have  no 
explicit  fiscal  responsibility  for  performing  the  V&V  as  the  instruction  is  now  written. 
If  members  of  the  TAT  belong  to  a  parent  organization  that  might  be  tasked  to 
perform  the  V&V,  there  could  be  additional  motivation  to  amplify  V&V  requirements 
to  obtain  additional  funding  for  their  parent  organization. 

Another  possible  impact  of  the  proposed  Navy  process  is  an  expanding  requirement 
for  intermediate  briefs  and  reviews  of  accreditation  proposals.  If  the  STAD  and  TAT 
evolve  into  an  apparent  extension  of  N81,  and  the  organization  preparing  an 
accreditation  package  is  organizationally  separated  by  several  layers,  numerous 
intermediate  reviews  of  the  accreditation  package  could  be  required.  These  reviews 
and  possible  revisions  of  the  package  could  cost  additional  money  and  time  beyond 
that  required  solely  by  the  draft  instruction. 

The  proposed  Navy  process  addresses  the  desirability  of  performing  V&V  in  parallel 
with  model  development  as  new  models  are  created.  Although  the  extent  of  such 
V&V  is  not  described  fully,  it  will  probably  yield  a  set  of  documents  that  provide 
information  on  the  model’s  assumptions,  limitations,  conceptual  description, 
verification  test  results,  any  desk  check  results  and  validation  results.  In  no  way  can 
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the  newly  developed  model  be  certified  by  the  TAT  until  the  application  is  known  and 
a  set  of  acceptance  criteria  developed.  If  the  TAT  attempts  to  certify  a  model 
without  knowing  the  intended  application  and  associated  acceptance  criteria,  they 
will  have  a  tendency  to  seek  near  perfect  correlation  between  the  model  and  test 
data.  This  tendency  would  tend  to  force  the  cost  of  model  development  to  be 
greater  than  necessary.  Even  if  they  obtained  extensive  validation  results,  a  critical 
element  of  the  application  might  be  overlooked  and  the  model  might  not  meet  all  the 
required  acceptance  criteria.  In  such  a  case,  model  usage  might  lead  to  invalid 
study  predictions  or  conclusions. 

The  four  different  levels  of  accreditation  defined  in  the  supplementary  document  on 
VV&A  principles  have  different  VV&A  requirements.  The  document  does  not  provide 
any  guidelines  for  correlating  the  accreditation  levels,  the  degree  of  acceptable  risk, 
the  potential  impacts  of  the  model  employment,  and  the  information  needed  to 
support  accreditation  at  each  level.  Without  such  guidelines,  the  mere  definition  of 
different  accreditation  levels  can  lead  to  confusion  in  defining  acceptance  criteria  for 
each  application. 

4.5  Air  Force  Accreditation  Policy  Description 

The  Air  Force  is  in  the  process  of  developing  a  policy  governing  VV&A  of  models 
and  simulations.  A  draft  document  dated  25  October  1993  outlines  the  emerging  Air 
Force  policy  concepts  governing  M/S  VV&A.  Certainly,  the  details  of  the  VV&A 
process  may  change.  However,  the  basic  concepts  are  expected  to  form  the  basis 
of  the  final  policy  that  is  implemented. 

The  emerging  Air  Force  policy  defines  eight  criteria  against  which  a  model  can  be 
compared.  Using  these  criteria,  M/S  used  in  Air  Force  studies  are  categorized  into 
three  groups.  The  principle  criteria  for  placing  an  M/S  into  one  of  the  three  groups 
are  completeness  of  documentation  and  community  acceptance  as  indicated  by  the 
extent  of  completed  accreditation.  According  to  the  draft  policy  this  categorization 
provides  an  integrated  priority  list  (IPL)  so  that  M/S  in  category  I  will  undergo  VV&A 
prior  to  category  II  and  category  III.  Category  II  models  will  undergo  VV&A  prior  to 
category  III.  Of  the  five  SMART  models,  ESAMS  and  RADGUNS  fall  into  category  I, 
ALARM  is  in  category  II,  AASPEM  is  in  category  III,  and  TRAP  is  not  included. 

The  Air  Force  VV&A  process,  depicted  in  Figure  5,  begins  with  a  determination  of 
the  level  of  V&V  that  is  required.  This  determination  is  made  by  a  committee 
composed  of  accreditation  proponents  (model  users),  model  developers, 
configuration  and  V&V  managers,  and  an  AF/XOM  representative.  The  committee 
decides  on  a  V&V  approach,  estimates  the  funding  required,  and  estimates  the 
personnel  required.  If  sufficient  documentation  does  not  exist,  a  minimum  set  of 
documents  must  be  developed  before  completing  the  V&V  effort. 

Once  the  V&V  requirements  are  defined,  the  accreditation  proponent  implements  the 
recommendations  of  the  committee.  The  committee  reconvenes  upon  completion  of 
the  V&V  effort  to  review  results  and  compare  the  amount  of  V&V  done  to  the  original 
plan.  They  provide  an  independent  technical  assessment  of  the  model,  a 
comparison  to  the  actual  versus  planned  V&V,  and  a  statement  of  risks  if  a  portion  of 
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the  V&V  process  was  omitted.  These  committee  findings  are  used  by  each 
accrediting  official  to  support  accreditation  of  the  model  for  each  particular 
application. 
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FIGURE  5  AIR  FORCE  VV&A  PROCESS 


According  to  Air  Force  policy,  each  M/S  will  have  a  single  V&V  focal  point  to 
coordinate  current  V&V  activities  and  to  serve  as  a  repository  for  information 
gathered  from  previous  V&V  efforts. 

4.6  Air  Force  Accreditation  Policy  Features 

The  Air  Force  process  is  simpler  than  either  the  Army  or  Navy  processes.  There  is 
no  defined  requirement  for  special  reports  to  document  VV&A  plans  or  reports. 

The  major  shortcoming  of  the  Air  Force  process  is  the  lack  of  guidance  on  the 
criteria  that  should  be  used  to  determine  the  amount  of  V&V  required.  Without  this 
guidance,  each  committee  will  base  their  recommendations  on  their  own  subjective 
judgment.  Under  these  conditions,  there  is  a  greater  likelihood  that  unnecessary 
V&V  might  be  required  to  ensure  that  the  committee  will  not  be  criticized  for 
recommending  a  model  for  accreditation  when  it  doesn’t  adequately  represent  the 
real  world.  Some  guidance  is  needed  on  how  acceptance  criteria  should  be 
developed  and  used  to  define  V&V  requirements. 
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A  second,  albeit  less  severe,  shortcoming  of  the  Air  Force  policy  is  the  requirement 
to  make  decisions  by  committee.  The  time  and  effort  required  to  constitute,  convene 
and  reconvene  several  committees  for  the  review  of  several  M/S  will  demand 
significant  resources.  Additionally,  the  problems  of  getting  key  officials  to  devote 
sufficient  time  to  these  committee  duties  and  to  actually  evaluate  model  V&V  results 
can  lead  to  committee  member  substitutions  and  lack  of  continuity  from  one  phase 
to  the  next.  Such  personnel  changes  can  result  in  additional  work  to  satisfy 
changing  V&V  requirements. 

The  Air  Force  concept  of  categorizing  M/S  into  an  integrated  priority  list  (IPL)  for 
VV&A  implies  that  once  a  model  has  undergone  the  prioritized  VV&A  and  has 
received  XOM  certification  it  is  suitable  for  use  in  all  Air  Force  studies.  If  this 
perception  is  valid,  the  Air  Force  philosophy  differs  from  the  other  two  services  and 
from  the  M/S  community  in  which  accreditation  must  be  performed  for  each 
application.  The  other  two  services  have  stated  that  domain  or  class  of  application 
accreditation  is  not,  by  itself,  sufficient  for  application  specific  accreditation. 

4.7  J-MASS  VV&A  Policy  Description 

Besides  the  three  services,  the  J-MASS  program  has  drafted  a  set  of  VV&A 
guidelines  that  are  intended  for  use  on  the  threat  models  developed  under  their 
program.  Since  the  J-MASS  architecture  will  be  used  for  development  of  new  M/S, 
and  the  SMART  products  should  support  VV&A  of  future  models  as  well  as  current 
ones,  the  J-MASS  VV&A  guidelines  were  analyzed  as  part  of  this  study. 

The  draft  J-MASS  guidelines  are  based  on  the  principle  that  V&V  is  done 
concurrently  with  model  development.  Accreditation  is  recognized  as  being  done 
after  model  development  is  complete  and  a  specific  purpose  has  been  identified  for 
the  model's  use.  The  J-MASS  VV&A  process  is  diagrammed  in  Figure  6. 

The  J-MASS  process  is  based  on  MIL- STD-21 67A  requirements  and  is  similar  to  the 
V&V  process  described  in  the  Navy  guidelines  for  V&V  of  new  models.  The  unique 
aspect  of  the  J-MASS  process  is  that  it  focuses  on  verification  and  validation  of 
modeled  threat  systems;  the  J-MASS  V&V  process  does  not  address  the  other 
aspects  of  the  models  such  as  target  algorithms,  environmental  features,  blue  force 
systems,  etc.  The  V&V  results  are  reviewed  by  the  Defense  Intelligence  Agency 
(DIA)  or  its  designee.  The  purpose  of  J-MASS  V&V,  including  DIA  review,  is  to 
ensure  that  the  models  accurately  represent  real-world  threat  system  performance. 
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FIGURE  6  J-MASS  VV&A  PROCESS 
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The  J-MASS  verification  process  is  performed  by  a  verification  team  that  reports  to 
the  responsible  approval  authority,  who  is  generally  the  program  manager  for  a 
given  model  development  effort.  Verification  involves: 

-  reviews  of  model  requirements  to  ensure  compatibility  with  an  intelligence 
baseline; 

-  reviews  of  the  engineering  design,  software  design,  and  test  plan  for 
completeness  and  consistency  with  J-MASS  standards; 

-  reviews  of  software  coding  and  testing  for  elements,  assemblies,  player  level 
components,  and  external  interfaces  to  ensure  completeness,  correctness, 
and  consistency  with  J-MASS  standards; 

-  reviews  of  model  documentation,  including  the  Model  Requirements 
Documentation  Manual,  to  ensure  that  it  is  complete,  correct,  consistent  and 
a  suitable  baseline  for  succeeding  software  development  activities; 

-  reviews  of  Software  Test  Plans  and  Test  Descriptions  to  ensure  that  these 
tests  demonstrate  all  system  and  software  requirements,  boundary  condition 
coverage,  and  input/output  condition  coverage; 

-  reviews  of  the  validation  matrix  to  avoid  duplication  of  effort; 

-  determination  that  internal  configuration  control  is  maintained  and  adequate 
records  of  test  results  are  maintained; 

-  preparation  of  the  verification  report. 

The  J-MASS  validation  effort  is  focused  on  development  of  two  documents:  the 
Design  Specification  Letter  (DSL)  and  the  Model  Validation  Report  (MVR).  The  DSL 
documents  the  model  specification  to  ensure  the  threat  baseline  will  support 
simulator  development.  It  also  documents  any  differences  between  the  threat  data 
and  the  specification.  The  MVR  documents  the  comparison  between  the  threat  and 
the  model. 

The  validation  effort  encompasses  the  following  steps: 

-  reviews  of  the  Threat  Definition  Document  to  ensure  that  it  addresses  all 
applicable  threat  characteristics; 

-  preparation  of  a  validation  matrix,  a  minimum  set  of  validation  criteria,  and  a 
list  of  planned  tests  to  sufficiently  stress  the  model  without  excessive 
dedicated  testing 

-  comparison  of  model  test  data  with  the  threat  baseline  to  determine  degree  of 
correlation; 

-  preparation  of  the  MVR  to  document  comparison  results. 

Both  the  draft  DSL  and  the  draft  MVR  are  sent  to  the  model  developer  for  comments 
and  possible  model  revisions.  When  finalized,  the  DSL  and  MVR  are  sent  through 
the  appropriate  service  channels  to  the  Chairman  of  the  Validation  Subcommittee  of 
the  CROSSBOW-S  Committee.  The  subcommittee  reviews  the  validation  reports 
and  develops  a  recommendation  for  approval  or  additional  development/validation 
work.  These  recommendations  are  passed  to  the  CROSSBOW-S  committee,  which 
reviews  them  and  passes  the  reports  to  the  DoD  Executive  Committee  on  Threat 
Simulators  (EXCOM).  Upon  recommendation  of  the  CROSSBOW-S  committee,  the 
EXCOM  chairman  will  generally  approve  the  validation  report.  If  there  is  any 
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disagreement  between  the  service  and  the  CROSSBOW-S  committee,  the  EXCOM 
will  meet  to  determine  what  action  should  be  taken. 

The  accreditation  effort  closely  parallels  the  validation  effort.  The  user  requirements 
are  determined  and  compared  with  the  validation  report  results  to  determine  if  they 
are  completely  addressed.  If  not,  an  accreditation  matrix,  incorporating  the  latest 
intelligence  data,  is  developed.  Additional  validation  is  performed,  and  the  results 
compared  to  the  requirements.  The  results  are  reported  in  an  accreditation  report 
that  is  presumably  sent  to  the  accrediting  authority.  Figure  7  shows  the 
accreditation  process  for  a  given  application  once  the  V&V  has  been  completed. 

4.8  J-MASS  VV&A  Policy  Features 

As  indicated  in  the  discussion  above,  both  verification  and  validation  are  performed 
by  a  team  that  is  independent  of  the  model  developer.  The  verification  tests  and 
validation  matrix  are  coordinated  to  eliminate  duplication  where  possible.  These 
features  contribute  to  V&V  efficiency.  The  verification  results  are  reported  to  the 
program  manager  and  the  developer  through  “quick  look”  and  draft  reports.  This 
closely  coordinated  relationship  is  beneficial  in  that  the  verification  results  are 
reviewed  and  acted  upon  with  a  minimum  of  excess  paperwork  and  reviews  by 
intervening  management  layers. 

In  contrast  to  the  verification  process,  the  validation  report  receives  multiple  reviews 
outside  the  developing  organization.  In  fact,  the  J-MASS  process  requires  that  the 
validation  reports  be  passed  to  the  CROSSBOW-S  committee  and  the  EXCOM 
through  the  services.  Therefore,  the  intermediate  reviewers  can  require  repeated 
report  modifications  and/or  additional  validation  work  to  be  accomplished  before 
ever  having  the  report  reviewed  by  the  approving  authority.  This  feature  can  result 
in  unnecessary  and  excessive  work. 

Another  aspect  of  requiring  CROSSBOW-S  and  EXCOM  approval  of  the  validation 
report  is  the  perception  that  they  are  determining  if  a  model  is  suitable  for  use.  The 
validation  report  describes  the  degree  of  correlation  between  the  model  and  the  DIA 
baseline  data.  When  EXCOM  approves  the  validation  report,  they  must  use  their 
own  criteria  for  judging  acceptability.  If  they  do  not  know  the  actual  accreditation 
requirements,  it  is  likely  that  they  would  err  on  the  conservative  side  and  use 
excessively  demanding  acceptance  criteria  to  avoid  criticism.  Such  requirements 
could  possibly  result  in  extensive  work  to  develop  a  model  that  has  a  very  high 
degree  of  correlation  with  the  threat.  This  effort  may  be  unnecessary  since  the 
accreditation  authority  establishes  the  acceptance  criteria  for  each  intended 
application  and  these  criteria  may  not  be  as  demanding  as  that  set  by  EXCOM. 

The  J-MASS  V&V  process  focuses  on  threat  systems.  Other  aspects  of  the  model 
are  not  addressed  in  the  draft  guidelines.  Even  if  the  process  is  followed  exactly, 
and  EXCOM  approval  is  granted,  a  model  user  will  probably  need  to  perform 
additional  V&V  on  the  parts  of  the  model  that  represent  blue  systems  and 
environmental  features. 
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FIGURE  7  J-MASS  ACCREDITATION  PROCESS 
5.  ANALYSIS  OF  ACCREDITATION  PRACTICES 


Although  the  Army  has  a  formal  regulation  in  place  that  defines  the  procedures  for 
accrediting  M/S,  and  the  other  services  are  developing  similar  policies,  our  interview 
results  indicate  that  the  organizations  actually  using  M/S  do  not  conform  to  the 
details  of  these  established  or  emerging  policies  in  most  cases.  Many  M/S  users 
consider  VV&A  as  part  of  the  overall  study  process,  and  integrate  VV&A  efforts 
within  the  context  of  the  total  study.  The  actual  practice  of  performing  a  study  while 
concurrently  building  model  credibility  is  described  in  the  following  paragraphs. 

5.1  Typical  Accreditation  Practices 

In  analyzing  the  information  in  the  interview  summaries,  some  similarities  emerged 
regarding  the  practices  used  by  the  different  organizations  to  select,  accredit,  and 
employ  a  model  in  a  particular  study  application.  Considering  these  similarities,  the 
varying  practices  can  be  classified  into  three  typical  types:  an  informal  procedure,  a 
semi-formal  procedure,  and  a  formal  procedure.  To  compare  and  contrast  these 
various  practices,  some  common  terms  were  needed  to  represent  various  officials  in 
the  different  organizations.  The  first  common  term  is  the  study  agent.  This  official 
establishes  study  requirements  and  directs  its  accomplishment.  Typically,  the  study 
agent  might  be  a  program  manager  or  the  head  of  a  study  agency  (such  as  AFSAA 
or  CNA)  which  is  tasked  by  some  higher  echelon  organization  to  perform  a  study. 
The  study  team  leader  is  the  senior  analyst  who  directs  a  team  of  analysts  in 
conducting  the  actual  study. 
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The  first,  and  simplest,  of  the  three  typical  practices  is  the  informal  procedure 
depicted  in  Figure  8.  This  procedure  is  usually  employed  by  organizations  that  have 
the  following  common  characteristics:  1)  they  are  staffed  with  experienced  analysts; 
2)  they  are  routinely  and  repeatedly  tasked  to  perform  COEAs  or  other  similar 
studies  in  support  of  acquisition  decisions;  and  3)  their  study  results  are  frequently 
needed  in  a  short  time.  The  major  feature  of  this  informal  practice  is  the  significant 
level  of  responsibility  placed  on  the  study  team  leader.  The  team  leader  has  almost 
complete  autonomy  in  selecting  the  model,  finding  and  validating  data,  modifying  the 
model,  and  determining  that  the  model  will  yield  credible  results  for  the  problem  at 
hand.  The  team  leader  reviews  the  study  product  and,  if  the  results  appear 
reasonable,  passes  them  along  to  the  study  agent  who  commissioned  the  study. 

The  next  typical  practice  is  generally  used  in  those  studies  where  more  time  is 
available  and/or  greater  DoD  visibility  is  expected.  In  this  semi-formal  procedure, 
depicted  in  Figure  9,  the  study  team  performs  essentially  the  same  steps  as  are 
performed  in  the  informal  procedure.  The  only  difference  is  that  the  study  agent  has 
an  opportunity  to  review  and  approve  the  study  plan.  This  study  plan,  which  usually 
outlines  the  proposed  steps  for  building  model  credibility,  is  submitted  after  tentative 
model  selection  and  identification  of  valid  source  data.  Once  the  plan  is  approved, 
the  study  team  completes  the  planned  steps  as  shown  in  the  diagram.  If  required  by 
the  study  plan,  a  formal  V&V  report  is  prepared. 

f  Study  Agent  J  Study  Team  |fe-. 


FIGURE  8  INFORMAL  PROCEDURE 
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FIGURE  9  SEMI-FORMAL  PROCEDURE 

The  formal  procedure  is  typically  employed  by  those  organizations  governed  by  an 
approved  VV&A  policy  document,  either  local  or  service  wide.  This  procedure, 
depicted  in  Figure  10,  is  more  structured  and  includes  a  formal  model  accreditation 
step.  The  study  agent  has  the  opportunity  to  approve  the  overall  study  plan  and  also 
to  accredit  the  model  based  on  a  formal  V&V  review.  However,  due  to  time 
constraints  usually  experienced  in  most  studies,  model  V&V  typically  takes  place  in 
parallel  with  the  actual  study.  The  formal  accreditation  is  often  accomplished  just 
before,  or  concurrent  with,  the  completion  of  the  study.  This  concurrency  forces  a 
close  coordination  between  the  V&V  effort  and  the  study  effort,  so  that  any  model 
deficiencies  that  might  be  uncovered  in  the  V&V  process  are  properly  corrected 
before  the  final  study  results  are  produced. 
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FIGURE  10  FORMAL  PROCEDURE 

The  activities  that  employ  one  or  another  of  the  above  procedures  are  identified  in 
Table  3.  Some  correlation  was  noted  between  the  type  of  agency  or  activity  and  the 
level  of  formality  in  the  practices  they  employ.  The  army  has  promulgated  a  policy 
requiring  formal  accreditation.  Therefore,  army  agencies  tend  to  employ  a  formal 
procedure.  Other  agencies  are  governed  by  local  policies  and  practices  that  have 
evolved  through  usage. 

Study  agencies  such  as  AFSAA  and  CNA,  having  highly  qualified  analysts  who 
frequently  perform  short  term  studies,  typically  use  the  informal  procedure.  AMSAA 
is  an  exception.  In  accordance  with  AR  5-11,  they  formally  accredit  those  models 
that  they  normally  use. 

The  two  acquisition  programs  included  in  this  investigation  follow  different  practices 
according  to  the  individual  program  manager's  guidelines.  The  Tomahawk  Program 
Office  recently  issued  an  instruction  requiring  a  formal  accreditation  for  all  M/S  used 
in  the  Tomahawk  program.  However,  the  instruction  is  so  new  that  formal 
accreditation  procedures  are  not  yet  in  place.  Until  now,  they  have  followed  the 
informal  practice.  The  Advanced  Strategic  and  Tactical  Expendables  (ASTE)  project 
undertook  a  formal,  independent  V&V  of  their  simulation  in  an  attempt  to  accredit  it. 
Those  V&V  results  are  not  yet  available,  and  the  efficiency  of  that  process  cannot  be 
evaluated. 
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TABLE  3  ACTIVITIES  USING  TYPICAL  PROCEDURES 


TYPE  OF 
ORGANIZATION 

INFORMAL 

PROCESS 

SEMI-FORMAL 

PROCESS 

FORMAL 

PROCESS 

Study  Agencies 

CNA 

AFSAA 

AMSAA 

Program  Managers 

PMA-281 

ASTE 

Operational  Test 
Organizations 

OPTEC 

OPTEVFOR 

AFOTEC 

Fleadquarters 

Organizations 

ASC 

ADC  SOPS  (1) 

NAVAIR-526 
ADC  SOPS 

ASC  (2) 

(1 )  ADC  SOPS  uses  this  procedure  only  if  time  constraints  die 

(2)  ASC  uses  this  procedure  for  large  and  complex  studies. 

tate. 

All  three  OT  organizations  formally  accredit  their  models.  OPTEC  complies  with  AR 
5-11.  AFOTEC  has  a  command  instruction  requiring  model  accreditation. 
OPTEVFOR  has  established  an  M/S  division  and  is  increasing  its  staff  to  perform 
formal  accreditations.  In  the  meantime,  they  have  requested  CNA  to  accredit 
models  for  OPTEVFOR  use.  It  was  noted  that  these  models  are  not  accredited  for  a 
specific  purpose,  but  for  operational  evaluation  in  general.  CNA  (not  OPTEVFOR) 
develops  the  acceptance  criteria  for  judging  model  suitability. 

The  last  group  of  organizations,  the  Air  Force  ASC,  NAVAIR  526,  and  ADC  SOPS, 
follow  either  informal  or  semi-formal  practices.  NAVAIR  526  nearly  always  submits 
an  analysis  plan  for  approval  before  beginning  a  study.  The  only  exception  is  when 
a  very  short  term  study  is  needed.  The  other  two  organizations  follow  either  process 
depending  on  the  time  available  and  the  expected  visibility  of  the  results.  As 
described  in  the  applicable  interview  reports,  these  organizations  generally  perform 
COEAs  much  shorter  in  duration  than  studies  performed  by  the  analysis  and 
operational  testing  activities.  Therefore,  they  seldom  have  the  time  to  accredit  their 
models  for  each  application. 

5.2  Features  of  the  Typical  Practices 

Our  analysis  shows  that  organizations  with  similar  missions  tend  to  adopt  similar 
VV&A  practices.  The  OT  organizations,  motivated  by  a  desire  to  substantiate 
conclusions  about  a  system  under  test,  all  strive  to  ensure  that  M/S  results  are 
sound:  they  usually  follow  the  formal  VV&A  procedure.  The  study  agencies,  having 
extensive  M/S  experience,  trust  their  analysts  to  be  the  best  judges  of  model 
credibility.  They  typically  employ  the  informal  procedure.  AMSAA  is  the  exception. 
Guided  by  AR  5-11,  they  have  switched  to  using  the  formal  procedure.  Even  so, 
they  maintain  a  balance  between  resource  demands  and  the  degree  of  VV&A 
formality.  They  make  extensive  use  of  briefings  and  annotated  briefing  charts  to 
avoid  the  cost  and  time  associated  with  the  preparation  and  review  of  formal  report 
documents.  The  remainder  of  the  organizations,  unless  driven  by  an  established 
policy,  balance  the  need  for  VV&A  formality  with  resource  constraints.  As  indicated 
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in  Table  3,  organizations  including  ASC  and  ADC  SOPS  will  choose  between 
different  levels  of  formality  depending  on  the  potential  visibility  of  the  study  results. 

Many  of  those  surveyed  in  this  study  had  common  concerns  regarding  VV&A  costs 
and  time  requirements.  The  CNA  and  NAVAIR-526  representatives  expressed 
reservations  about  the  formal  VV&A  process  being  considered  by  the  Navy.  They 
felt  that  it  would  be  far  too  costly  and  time  consuming.  Mr.  Clayton  Thomas  of 
AFSAA  stated  that  the  planned  Air  Force  policy  should  be  as  “permissive  as  the 
traffic  will  bear  under  the  current  DoD  guidelines”  and  GAO  interest.  Fie  also 
recognized  that  time  constraints  sometimes  force  the  use  of  models  that  are  not 
totally  suitable.  These  statements  are  consistent  with  the  observations  that  most 
organizations  involved  in  performing  studies  use  processes  that  are  much  less 
formal  than  those  described  in  service  level  instructions  and  regulations.  Clearly, 
the  major  feature  of  the  typical  processes  is  the  balance  between  resource 
constraints  and  the  level  of  VV&A  formality. 

Some  activities,  though  concerned  about  M/S  validity  and  study  credibility,  felt  that 
competent  analysts  who  deal  with  M/S  on  a  regular  basis  are  best  suited  to  judge  if 
a  particular  M/S  is  acceptable  for  a  particular  application.  Even  if  a  formal  review 
process  is  followed  for  approving  a  study  plan  or  for  accrediting  a  model,  the  study 
analyst  has  primary  responsibility  for  generating  the  required  information  and 
presenting  it  to  the  reviewing  official.  It  was  recognized  that  an  analyst's  judgment 
on  M/S  suitability  can  be  affected  by  several  factors,  including:  1)  schedule 
constraints;  2)  a  possible  lack  of  models  that  meet  study  requirements;  or  3) 
individual  biases  of  the  analysts  toward  familiar  models.  In  spite  of  these  factors, 
agencies  that  routinely  perform  studies,  such  as  CNA  and  AFSAA,  have  groomed  a 
cadre  of  competent  analysts  who  have  merited  the  confidence  of  their  superiors.  In 
at  least  one  example,  their  competence  is  recognized  outside  their  own  organization. 
OPTEVFOR  enlists  the  services  of  CNA  to  perform  V&V  on  the  models  which 
OPTEVFOR  uses.  This  tasking  results  from  OPTEVFOR’s  perception  of  CNA’s 
credibility  in  dealing  with  models. 

One  common  discipline  followed  by  many  current  analysts  is  the  delineation  of 
acceptance  criteria  for  evaluating  model  acceptability.  AMSAA  has  included  them 
as  a  key  element  of  their  VV&A  checklist.  OPTEC  personnel  stressed  the 
importance  of  acceptance  criteria  for  accreditations,  which  are  always  application 
specific.  The  application  MOEs  or  MOPs  govern  development  of  the  acceptance 
criteria.  The  Air  Warfare  Center  interviewee  stressed  that  selection  of  the  MOEs  or 
MOPs  is  probably  the  most  critical  aspect  of  any  study.  If  improper  MOEs  or  MOPs 
are  selected,  the  study  results  can  be  totally  erroneous,  even  if  properly  accredited 
models  are  used.  Appropriate  and  acceptable  MOEs  or  MOPs  are  the  basis  for 
acceptance  criteria  definition.  These  criteria  should  be  used  to  judge  whether  a 
model  will  be  suitable  for  use  in  a  given  study  application.  These  criteria  should 
address  the  issue  of  the  required  fidelity  between  model  predictions  and  real  world 
results.  Such  fidelity  criteria  would  then  be  applied  to  validation  results  to  judge 
model  acceptability. 

Another  common  feature  that  characterizes  the  existing  VV&A  practices  is  the 
requirement  to  provide  specific  types  of  information.  Most  organizations  have 
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identified  the  information  required  to  determine  whether  the  model  is  acceptable. 
CNA’s  requirements  are  focused  on  verification  information,  such  as  results  of  logic 
checks  of  the  algorithms,  sensitivity  analyses’  results,  and  reviews  of  model 
assumptions  and  constraints.  AMSAA  has  a  checklist  of  information  elements  that 
are  reviewed  before  accreditation.  In  contrast,  the  official  Army  and  Navy  policy 
guidance  define  formal  processes  for  accreditation  but  do  not  specify  any  required 
information  that  must  be  reviewed.  Although  these  policy  documents  provide 
examples  of  the  types  of  information  that  can  be  reviewed,  they  do  not  establish  any 
minimum  baseline  information  that  must  be  checked. 

6.  COMPARISON  OF  ACCREDITATION  POLICIES  AND  PRACTICES 

The  accreditation  processes  defined  in  the  Army,  Navy,  and  J-MASS  policy 
documents  appear  to  be  excessively  bureaucratic  and  costly  when  compared  to  the 
current  practices.  Numerous  special  documents  must  be  produced  and  reviewed 
through  a  hierarchical  chain.  The  preparation  and  review  of  such  documentation 
can  lead  to  multiple  revisions,  and  make  these  processes  time  consuming  and 
costly.  The  draft  Air  Force  policy,  while  rather  simple  and  less  demanding  on 
resources,  still  requires  a  committee  to  determine  V&V  requirements.  Like  the  other 
service  policies,  the  draft  Air  Force  policy  does  not  provide  any  guidance  on  how  the 
V&V  requirements  (or  model  acceptance  criteria)  should  be  determined. 

A  major  disadvantage  resulting  from  the  lack  of  uniform  guidelines  for  establishing 
acceptance  criteria  is  the  potential  variability  in  requirements  resulting  from 
subjective  perceptions.  Without  uniform  guidelines  for  developing  V&V 
requirements,  different  reviewing  officials  can  impose  and  demand  V&V  to  be  done 
to  different  depths.  Consequently,  planning  and  budgeting  for  the  V&V  of  a  model 
will  be  extremely  difficult  unless  the  model  and/or  study  reviewers  are  known 
beforehand  and  consulted  during  the  planning  and  budgeting  process  to  determine 
the  expected  V&V  scope. 

The  requirement  to  have  a  team  of  technical  personnel  participate  in  the  VV&A 
review  processes  is  difficult  to  implement  efficiently.  Unless  team  membership  is  a 
full  time  responsibility,  team  members  will  be  assigned  on  a  collateral  duty  basis. 
Each  time  a  VV&A  review  or  activity  is  scheduled,  all  the  required  personnel  must  be 
assembled.  If  multiple  reviews  or  activities  are  required  as  part  of  the  VV&A  cycle 
for  a  single  model  for  a  single  acquisition,  the  same  team  members  will  normally  be 
required.  This  requirement  will  place  additional  burdens  on  team  coordinators  to 
schedule  the  necessary  reviews  expeditiously  and  yet  obtain  full  attendance. 
Policies  requiring  independent  technical  reviews  of  all  V&V  activities  will  probably 
lead  to  delays  in  accreditation  and  to  unnecessary  costs  to  schedule  and  assemble 
the  participants. 


27 


The  requirements  imposed  by  the  Army,  Navy,  and  J-MASS  for  multiple  independent 
technical  reviews  of  V&V  results  do  not  appear  justified.  The  advantage  of  having 
independent  reviews  is  to  ensure  that  uniform  technical  standards  are  used,  and  to 
negate  the  possibility  of  participant  bias.  The  drawback  is  the  difficulty  of  obtaining 
the  “right”  reviewers  (however  defined)  and  the  additional  time  and  cost  impacts  of 
orienting  team  participants  and  assembling  them  repeatedly  to  conduct  reviews. 
However,  if  a  set  of  acceptance  criteria  were  defined  for  each  application,  a 
journeyman  analyst  with  intellectual  honesty  could  exercise  judgment  as  to  whether 
a  model  met  the  criteria  or  not.  To  eliminate  errors  and  bias,  the  results  of  the 
comparison  could  be  made  available  for  management  review.  Thus,  the  need  for 
multiple  independent  reviews  could  be  replaced  by  the  less  burdensome  approach 
of  establishing  acceptance  criteria  for  a  model  and  allowing  the  involved  analysts  to 
develop  the  information  about  the  model  for  comparison  with  the  criteria.  The 
comparison  results  could  be  briefed  to  the  accrediting  official  as  a  basis  for 
accreditation. 

Another  point  of  comparison  is  the  degree  of  responsibility  placed  on  the  analyst.  In 
contrast  to  the  high  degree  of  confidence  placed  in  the  analysts  by  the  organization 
heads  in  their  current  practices,  the  service  and  J-MASS  policies  tend  to  denigrate 
the  analyst  and  place  more  credence  in  ad  hoc,  independent  teams  chartered  to 
perform  V&V  or  review  V&V  results.  The  structure  and  membership  of  these  teams, 
and  the  technical  requirements  for  assignment  to  the  teams,  are  not  generally 
addressed  in  the  instructions.  However,  because  the  requirement  for  independent 
V&V  teams  is  included  in  these  policy  guides,  it  can  be  assumed  that  the  policy 
makers  lack  confidence  in  either  the  technical  competence  or  the  integrity  of  the 
individual  organization’s  analysts. 

The  advantage  of  giving  analysts  and  not  independent  review  teams  the 
responsibility  for  defining  acceptance  criteria  is  best  seen  in  light  of  bureaucratic 
psychology.  If  an  independent  technical  reviewer  or  review  team  is  responsible  for 
determining  V&V  adequacy,  the  study  analyst  will  propose  a  V&V  plan  with  the  more 
likely  goal  of  obtaining  the  reviewer’s  concurrence  than  the  goal  of  fulfilling  objective 
criteria  suited  to  the  study.  The  plan  will  most  likely  be  structured  to  meet  all  the 
requirements  that  were  met  in  the  last  review  that  official  conducted.  Typically,  a 
previously  approved  V&V  plan  is  found  and  analyzed,  and  a  new  one  reflecting  the 
new  application  is  prepared.  Little  independent  thought  is  given  to  what  unique 
requirements  might  exist  for  the  new  application.  While  the  extent  of  the  V&V 
performed  may  be  uniform  from  one  application  to  the  next,  there  can  be  no 
confidence  that  the  unique  V&V  requirements  of  each  application  will  be  addressed. 
The  pseudo-standard  V&V  requirements  that  are  carried  over  from  one  plan  to  the 
next  may  be  either  inadequate  or  excessive  when  compared  to  an  objective  set  of 
requirements  derived  from  the  study  MOEs  and  MOPs. 

7.  SYNTHESIS  -  NOTIONAL  PROCESS  DESCRIPTION 

Current  practices  for  performing  a  study  are  relatively  streamlined  and  efficient.  The 
underlying  concept,  which  involves  defining  the  study  MOEs  or  MOPs,  deriving 
model  acceptance  criteria,  gathering  V&V  information  to  compare  with  the  criteria, 
and  accrediting  the  model  based  on  that  comparison,  appears  well-conceived  and 
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logical.  This  approach  tends  to  mitigate  potential  biases  by  requiring  that  application 
specific  accreditation  criteria  be  defined  before  model  selection.  The  major 
disadvantages  noted  in  these  procedures  are:  1)  an  inconsistent  application  of  this 
approach;  2)  the  potential  biases  of  the  analysts  coupled  with  resource  limitations 
that  might  cause  incomplete  data  gathering  and/or  improper  comparisons;  and  3)  a 
possible  lack  of  uniformity  and  experience  in  establishing  acceptance  criteria  for 
each  specific  application.  Any  policies  that  might  be  developed  to  improve  the 
quality  of  study  decisions  should  be  based  on  the  advantages  of  the  current 
practices  but  designed  to  overcome  their  disadvantages  while  conserving  scarce 
resources. 

Considering  these  goals,  a  notional  process  for  accreditation  of  M/S  should  have  the 
following  features: 

-  Minimal  requirements  for  VV&A  reports  unless  such  reports  can  be  used  as  a 
source  of  information  for  subsequent  accreditation  decisions; 

-  Decentralized  decision-making  authority  to  minimize  schedule  impacts  and  to 
permit  tradeoffs  between  risk  and  cost; 

-  Focus  on  quality  accreditation  decisions  by  providing  a  mechanism  or 
methodology  to  introduce  uniformity  in  establishing  comprehensive,  tailored 
acceptance  criteria  based  on  study  MOEs/MOPs. 

A  notional  process  that  extracts  the  best  features  of  both  the  service  policies  and  the 
actual  practices  is  depicted  in  Figure  11.  This  process  addresses  the  total  study 
evolution,  not  just  the  accreditation  process.  In  this  process,  the  study  team  would 
analyze  the  problem  at  hand,  define  the  MOEs  or  MOPs,  develop  the  acceptance 
criteria,  and  plan  the  necessary  V&V  and  study  activities.  The  study  agent  would 
review  the  plan,  paying  close  attention  to  the  MOEs/MOPs  and  acceptance  criteria. 
If  in  doubt  about  either’s  adequacy,  a  separate  team  of  expert  analysts  would  be 
available  to  assist  in  this  review.  This  expert  team  would  help  ensure  that  proper 
MOEs  or  MOPs  were  selected  and  a  complete  set  of  acceptance  criteria  had  been 
defined.  The  team  of  expert  analysts  could  be  chartered  by  either  DoD  or  the 
individual  services. 

The  primary  benefits  of  this  approach  derive  from  the  “principle  of  subsidiarity,” 
which  delegates  no  decision-making  function  or  action  to  a  higher  level  that  can  be 
successfully  completed  at  a  lower  level.  This  places  the  burden  of  professional 
integrity  squarely  on  the  shoulders  of  those  required  to  generate  the  product,  and 
takes  maximum  advantage  of  local  expertise  most  familiar  with  the  problem  at  hand. 
Administrative  overhead  is  reduced  because  the  technical  requirements  of  decision¬ 
making  authorities  reduce  to  a  comparison  of  already  defined  acceptance  criteria 
(agreed  to  beforehand)  with  the  demonstrated  results  of  V&V.  Users  are 
enfranchised  in  the  process  and  decision  makers  are  assured  that  those  most 
familiar  with  the  problem  are  responsible  for  the  results. 
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FIGURE  1 1  ANALYTICAL  PROCESS  WITH  V&V  PLAN  APPROVAL 

To  aid  the  effectiveness  of  this  process,  study  analysts  should  be  experienced  in 
defining  MOEs  or  MOPs  for  analytical  studies,  familiar  with  a  wide  range  of 
applicable  M/S,  and  able  to  develop  appropriate  acceptance  criteria  from  this 
information.  The  modeling  community,  led  by  the  Defense  Modeling  and  Simulation 
Office  (DMSO)  should  develop  a  common  checklist  of  standard  questions  and 
issues  for  guidance  in  formulating  M/S  acceptance  criteria  in  support  of  studies. 
These  acceptability  criteria  should  address  issues  such  as  the  required  degree  of 
fidelity  between  the  model  results  and  real  world  data  as  a  function  of  intended 
accreditation  level,  the  envelope  within  which  these  comparisons  are  valid,  and  the 
total  envelope  in  which  the  model  can  be  expected  to  provide  realistic  results.  This 
envelope  should  be  derived  from  a  combination  of  V&V  results  and  sensitivity 
analyses.  These  two  factors  should  yield  the  range  of  input  parameters  over  which 
the  model  algorithms  remain  linear,  or  at  least  analytic. 

If  the  study  is  complex,  or  if  the  nature  of  the  study  precludes  a  clear  definition  of 
acceptance  criteria  before  the  analysis  is  performed,  the  process  could  be  modified 
to  allow  the  study  agent  a  second  review  of  finalized  acceptance  criteria  and  model 
V&V  results.  This  modified  process  is  shown  in  Figure  12.  If  needed,  the  expert 
analysis  team  would  be  available  to  assist  the  study  agent  in  determining  the 
technical  adequacy  of  the  acceptance  criteria  and  V&V  results.  The  second  review, 
which  is  essentially  an  accreditation  review,  would  lead  to  model  accreditation  or  to 
additional  model  modification  and/or  V&V.  If  the  model  or  the  MOEs/MOPs  are 
changed  due  to  the  V&V  or  the  accreditation  reviews,  the  analysis,  proceeding  in 
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parallel  with  the  V&V,  would  be  checked  to  ensure  that  the  changes  did  not  affect 
analytical  results  to  date. 
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FIGURE  12  ALTERNATE  ANALYTICAL  PROCESS  WITH  ACCREDITATION 

The  proposed  process  is  predicated  on  the  assumption  that  acceptance  criteria  are 
the  single  most  important  factor  in  determining  M/S  suitability  for  a  particular 
application,  and  that  with  these  criteria,  any  knowledgeable  person  or  group  can 
reach  essentially  the  same  decision  regarding  M/S  suitability.  This  also  assumes 
that  the  analysts  who  perform  the  studies  are  intellectually  honest.  If  they  cannot  be 
trusted,  then  an  independent  review  of  the  supporting  material  and  its  comparison 
with  the  criteria  might  be  necessary.  This  may  be  the  case  where  the  accreditation 
proponent  has  a  vested  interest  in  the  results  of  an  accreditation  decision.  Under 
these  circumstances  the  modified  process  would  be  most  applicable. 

In  these  notional  processes  the  team  of  expert  analysts  acts  as  an  independent 
“quality  assurance”  agent.  This  team  would  only  be  required  to  assist  in  certain 
circumstances  or  when  the  study  agent  desires  their  technical  help.  Therefore, 
excessive  resources  are  not  spent  in  mandated  accreditation  information  reviews  for 
simple,  straightforward  studies.  Besides  providing  review  functions  when  requested, 
the  expert  teams  could  also  benefit  the  study  analysts  by  developing  guidelines  and 
suggested  checklists  for  developing  both  MOEs/MOPs  and  model  acceptance 
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criteria.  Such  guides  would  facilitate  analyst  training  in  these  critical  areas  and  could 
be  used  as  a  checklist  for  study  agents  in  conducting  their  own  reviews  on  future 
studies. 

In  the  notional  processes,  the  study  team  is  primarily  responsible  for  collecting  V&V 
information,  performing  additional  V&V  as  necessary,  and  comparing  the  results  to 
the  acceptance  criteria.  In  the  absence  of  any  guidance  from  the  study  agent,  the 
study  team  leader  would  determine  the  depth  of  information  needed  to  support  a 
conclusion  that  the  model  meets  each  of  the  acceptance  criteria.  It  turns  out  that 
SMART  products  provide  a  primary  source  of  most  of  this  information,  as  discussed 
in  Volume  II  of  this  report  entitled  “Information  Requirements  in  Support  of 
Accreditation.” 

8.  INTEGRATION  WITH  SMART  V&V  PROCESS 

The  SMART  Project  is  developing  a  V&V  process  that  decomposes  a  model  into 
basic  functional  elements  and  collects  data  from  on-going  tests  that  can  be  used  to 
validate  common  functional  elements  in  several  different  models.  In  parallel,  each 
functional  element  is  verified  by  an  independent  agent.  The  process  includes  a 
sensitivity  analysis,  to  determine  the  most  critical  functional  elements,  and 
correlation  of  the  various  functional  element  validation  results  with  an  overall  model 
assessment. 

The  SMART  process  has  been  divided  into  a  series  of  incremental  steps  to  facilitate 
application  to  a  newly  selected  model  in  a  fiscally  reasonable  manner.  This 
incremental  application  philosophy  is  described  in  greater  detail  in  the  companion 
report,  Volume  II  entitled  “Information  Requirements  in  Support  of  Accreditation.”  A 
diagram  showing  the  incremental  steps  is  shown  in  Figure  13. 

The  incremental  application  of  the  SMART  V&V  process  to  a  given  model  will 
generate  the  normal  SMART  products  in  a  sequential  series  that  closely  matches 
the  information  requirements  at  the  different  phases  and  for  different  levels  of  the 
Navy  accreditation  process.  Table  4  shows  the  series  of  sequential  products  and 
how  they  can  be  grouped  into  accreditation  levels  and  phases  that  are  similar  to 
those  under  discussion  in  the  Navy  and  Air  Force.  As  part  of  the  continuing  SMART 
Project,  cost  estimating  parameters  for  each  incremental  step  are  being  developed. 

The  incremental  accreditation  approach,  if  adopted  by  all  users  of  a  model,  will 
permit  the  early  users  to  develop  the  basic  information  to  satisfy  low  risk  applications 
and  to  lay  the  foundation  for  more  in  depth  V&V  by  subsequent  users.  Obviously,  if 
a  new  model  is  used  initially  for  a  high  risk  study  and  in  depth  V&V  is  required,  the 
initial  user  will  be  faced  with  undertaking  the  entire  effort.  Even  in  this  case, 
however,  adoption  of  the  SMART  process  will  allow  the  user  to  access  and  use 
existing  test  data  that  has  been  collected  and  catalogued  for  validation  of  specific 
functional  elements. 
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FIGURE  13  INCREMENTAL  SMART  PROCESS 
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TABLE  4  SMART  PRODUCT  RELATIONSHIP  TO  ACCREDITATION  LEVELS 


INCREMENTAL 
V&V  PRODUCT 

PROCESS 

SUPPORT 

PRODUCT 

ACCREDITATION 

LEVEL 

ACCREDITATION 

PROCESS 

PHASE 

Acceptable  Model 
Documentation 

Level  1 

Phase  1 

Usage  &  V&V 
History  Report 

Refined  Template 

C/M  Requirements 
Report 

Level  2 

Phase  2 

Sensitivity  Analysis 
Report 

List  of  Assumptions 
&  Limitations 

C/M  Status  Report 

Logical  Verification 
of  Algorithms 

Face  Validation 
Report 

Test  Plan  Library 

Assessment  Plan 
Library 

PDDD 

Detailed  FE  Test 
Plans 

Levels  3  &  4 

Verification  Reports 

Reduced  Test  Data 

Test  Report 
Supplements 

Validation  Reports 
(FEAR  &  MAR) 

Assuming  that  a  model  has  been  subjected  to  the  SMART  process  and  a  set  of 
baseline  V&V  products  exist,  an  analyst  would  employ  them  in  selecting  a  model, 
planning  additional  V&V,  and  assembling  the  accreditation  information.  Figure  14 
shows  how  the  SMART  products  would  facilitate  execution  of  the  applicable  study 
tasks.  The  advantage  of  following  the  SMART  V&V  process  and  employing  the 
SMART  products  during  any  application  specific  V&V  is  that  any  new  V&V  results 
would  be  added  to  the  V&V  library  for  the  benefit  of  future  model  users.  In  this  way, 
each  user  would  have  the  benefit  of  in-depth  V&V  results  with  a  minimal  expenditure 
of  individual  project  resources. 
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FIGURE  14  SMART  PRODUCT  USAGE 


9.  SUMMARY  AND  RECOMMENDATIONS 

In  summary,  the  current  accreditation  policies  rely  on  extensive  bureaucratic  reports, 
reviews,  and  checks  that  will  most  likely  waste  resources  and  yet  will  not  necessarily 
result  in  more  suitable  models  for  each  application.  The  existing  practices  for  model 
acceptance  and  use  are  generally  efficient  but  suffer  from  a  lack  of  consistency 
across  organizational  lines.  An  approach  that  capitalizes  on  the  best  features  of  the 
current  practices  while  meeting  the  goals  of  the  service  policies  is  needed. 

Considering  that  the  most  important  concerns  in  current  model  selection  and 
accreditation  practices  are  proper  selection  of  study  MOEs/MOPs  and  subsequent 
derivation  of  model  acceptance  criteria  from  these  MOEs/MOPs,  a  suggested 
accreditation  process  that  incorporates  these  steps  has  been  synthesized.  This 
process  is  simple,  efficient,  and  focuses  on  defining  adequate  criteria  for  model 
accreditation  based  on  appropriately  selected  MOEs/MOPs.  The  process  is 
compatible  with  and,  in  fact,  made  more  practical  if  the  SMART  V&V  procedures  are 
used  to  obtain  the  V&V  information  needed  to  support  an  accreditation  decision. 
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Based  on  the  foregoing  analysis  the  following  recommendations  are  submitted: 


1.  DoD  and  service  accreditation  policies  should  focus  on  development  and 
use  of  complete  and  comprehensive  MOEs  or  MOPs  and  model 
acceptance  criteria  to  support  model  accreditation  for  particular 
applications.  Requirements  for  obligatory,  independent  reviews  and 
formal  reports  should  be  minimized. 

2.  DoD  and  service  agencies  responsible  for  M/S  policy  development  should 
charter  one  (or  a  few)  team(s)  of  expert  analysts  to  facilitate  development 
and  review  of  MOEs/MOPs  and  acceptance  criteria  when  requested. 

3.  DMSO  should  sponsor  development  of  initial  guidelines,  tailored  to  each 
class  of  applications,  that  will  guide  analysts  in  defining  their  own 
MOEs/MOPs  and  acceptance  criteria  for  each  application,  and  for  tailoring 
V&V  efforts. 

4.  DMSO  should  sponsor  a  demonstration  effort  in  which  the  notional 
accreditation  process  described  in  this  study  is  used  and  independently 
evaluated. 

5.  DMSO  should  seek  out  opportunities  to  create  automated  systems  that 
will  assist  analysts  in  defining  MOEs/MOPs  and  acceptance  criteria.  In 
essence  these  would  be  expert  systems  that  assist  in  defining  V&V 
requirements  tailored  to  each  application. 
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APPENDIX  B 


INTERVIEW  GUIDE 


ACCREDITATION  REQUIREMENTS  STUDY 
INTERVIEW  GUIDE 


1  For  what  purpose(s)  does  your  agency  use  M&S? 

2  What  is  your  accreditation  process? 

3  Does  it  conform  to  any  of  the  process  models  we  have  already  diagrammed? 
Which  one? 

4  Who  has  authority  to  accredit  models  for  your  use? 

5  Is  that  authority  defined  in  any  instruction  or  charter? 

6  If  so,  what  is  that  document? 

7  Is  that  instruction/charter  agency-specific  or  service-wide? 

8  Can  we  have  a  copy? 

9  On  what  basis  are  accreditation  decisions  made? 

10  Considering  the  various  VV&A  elements,  (e.g.  Data  source  verification,  logical 
verification,  code  verification,  comparison  with  test  results,  comparison  with  other 
model  results,  etc.)  which  ones  are  required  for  your  accreditation?  Which  ones 
are  desirable? 

1 1  Is  validation  of  any  sort  required  prior  to  accreditation? 

12  Considering  the  validation  elements  already  listed,  which  ones  are  done?  Are 
there  any  others? 

13  Which  elements  would  you  consider  most  beneficial? 

14  Must  the  validation  have  been  done  on  the  version  of  the  model  being  accred¬ 
ited? 

15  Are  special  tests  run  to  validate  a  model,  or  are  previous  test  data  acceptable? 

16  For  the  last  few  models  accredited,  what  was  the  cost  of  any  special  testing  or 
data  collection  which  was  done? 

17  Have  any  models  been  accredited  solely  on  the  basis  of  past  usage,  or  on  the 
basis  of  “face  validation”? 

18  What  agencies  do  you  interface  with  to  obtain  information  on  VV&A  history  of  the 
model  you  intend  to  accredit? 

19  Is  any  verification  required  prior  to  accreditation? 
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20  Considering  the  verification  elements  already  listed,  which  ones  are  done  to  sup¬ 
port  an  accreditation  decision? 

21  Are  there  any  others? 

22  Is  configuration  management  of  a  model  required  prior  to  accreditation? 

23  To  what  extent? 

24  Is  configuration  management  handled  in-house,  or  does  your  agency  rely  on 
configuration  management  activities  from  other  agencies? 

25  Describe  any  configuration  management  requirements  imposed  on  the  accredita¬ 
tion  process,  and  any  procedures  used  to  fulfill  them. 

26  How  are  any  validation,  verification,  or  accreditation  activities  documented? 

27  Are  these  activities  and  results  published  in  any  form  by  which  other  model  users 
can  become  informed  of  the  results  and  use  them? 

28  What  are  these  forms,  and  can  we  have  access  to  some  typical  accreditation  re¬ 
ports? 

29  What  types  of  model  information  is  needed  to  determine  whether  the  model  ful¬ 
fills  the  analytical  requirement?  What  model  summary  data  would  you  want  in  a 
database? 

30  Do  you  have  any  examples  of  criteria  which  you  used  as  a  basis  for  making  an 
accreditation  decision  or  in  determining  that  a  model  was  acceptable  for  a  par¬ 
ticular  use? 

31  In  looking  at  the  matrix  of  accreditation  requirements,  which  ones  are  either  re¬ 
quired,  desired,  or  considered  acceptable  to  accredit  a  model. 

32  Who  else  in  this  agency  can  we  talk  to  concerning  accreditation  efforts? 

33  Who  else  in  your  service  should  we  be  talking  to  about  accreditation  require¬ 
ments? 
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AIR  FORCE  SPECIFIC  QUESTIONS 


For  AFOTEC 

34  Two  guides  exist  for  AFOTEC  M&S  accreditation;  POC:OAN  Draft  of  5  Feb.  92 
and  the  LG4  guide  of  Jan.  91.  Which  one(s)  express  AFOTEC  policy?  Who  is 
Accrediting  authority,  M/S  Committee  or  as  appointed  in  Accreditation  Plan? 

35  LG4  guide  says  analysts  are  responsible  for  Configuration  Control  for  M&S  they 
develop.  Does  AFOTEC  do  any  CM  for  existing  models? 

36  When  using  existing  models,  does  AFOTEC  require  that  the  four  documents 
(Management  manual,  Analysts  Manual,  Programmers  Manual,  &  Users  Manual) 
be  available? 

37  For  existing  models  if  manuals  do  not  meet  your  standards,  are  they  rewritten? 

38  Should  the  DOT&E  issues  (identified  on  the  separate  issues  sheet)  be  addressed 
or  investigated  as  part  of  the  SMART  process? 


NAVY  SPECIFIC  QUESTIONS 

39  What  relationship  exists  between  the  Navy  VV&A  plan  drafted  under  SPAWAR 
sponsorship  and  the  JACG  initiative  to  develop  a  unified  VV&A  process  for  sur¬ 
vivability  models? 

40  Of  the  SMART  model  set,  only  ESAMS  is  listed  in  the  M&S  catalogue  being  pre¬ 
pared  by  SPAWAR.  Should  the  others  be  included? 

41  Goal  #10  for  Team  MIKE  is  to  "establish  a  methodology/procedure  for  providing 
documentation,  certification,  and  configuration  management  for  Naval  Warfare 
models."  Do  the  SMART  documentation,  assessment  reports,  and  CM  plans 
meet  this  requirement? 

42  Goal  #7  is  to  "establish  and  catalog  common  data  bases  IAW  the  principles  sup¬ 
porting  Navy  evolution  toward  common  data  bases  through  the  NWTDB  process, 
set  forth  in  OPNAVINST  9410.5."  Need  more  information  about  that  instruction 
and  related  Navy  policies  on  data  base  development. 
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ARMY  SPECIFIC  QUESTIONS 


43  Is  there  an  Organization  Chart  which  shows  the  various  Army  organizations, 
committees,  and  activities  which  get  involved  in  accreditation? 

44  Any  contacts  in  Army  Model  and  Simulation  Management  Office  (AMSMO), 
Office  Deputy  Chief  of  Staff  for  Operations  (ODCSOPS),  or  MISMA  who  should 
be  interviewed? 

45  For  MISMA  What  progress  has  there  been  on  the  MISMA  master  plan 
(referenced  in  the  Jim  Metzger  brief)  for  credibility  assessment?  Is  that  AR  5-1 1  ? 

46  For  AMSMO  What  progress  on  the  AMSMO  "How  to"  handbook?  Who  is 
preparing  it?  Can  we  contact  them  for  information  gathering  purposes?  Who  is 
Point  of  Contact  (POC)? 

47  Are  accreditation  procedures  listed  on  pg.  6  of  SAUS-OR  memo  dated  30  Oct  89 
still  complete  &  valid?  What  does  the  statement  "Review  of  how  input  data  and 
scenario  data  are  used  or  modified  internal  to  the  model. "  mean? 

48  Can  I  get  a  sample  of  a  "good"  V&V  plan  and  V&V  report? 

49  For  OPTEC  Is  the  V&V  documentation  format  (end.  6  of  OPTEC  IPG  92-2)  still 
valid?  What  is  an  "uncertainty  analysis"? 

50  Are  any  SMART  models  included  in  the  list  of  Army  "Touchstone"  M&S?  What 
aircraft  survivability  models  are  included? 

51  AR  5-1 1  has  provisions  for  accreditation  of  a  model  for  a  "Class  of  Applications". 
Define/describe  a  "class  of  applications". 

52  Should  SMART  attempt  to  produce  a  V&V  report  supporting  accreditation  for  a 
Class  of  Applications?  If  so  what  class(es)  should  be  covered? 

53  Should  the  SMART  produced  test  data  conform  to  AR  25-9  and  the  Army  Data 
Encyclopedia?  If  so,  what  are  the  requirements  and  costs?  Would  Army  provide 
funds  to  support  tailoring  data  to  these  requirements?  Who  should  funding  re¬ 
quests  of  this  nature  be  directed  to? 

54  What  are  some  typical  acceptability  criteria  which  are  used  in  evaluating  a  model 
for  a  class  of  applications? 

55  Where  can  I  get  a  copy  of  AR  25-1  and  AR  25-9? 
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APPENDIX  C 

INTERVIEW  SUMMARIES 
ARS  INTERVIEW  REPORT 


INTERVIEWEES:  Mr.  Wyoming  (Duke)  Paris  DATE:  1  April  1993 
Mr.  Jeff  Gavlinski 
MAJ  Mike  Barton 
Mr.  Phil  Beavers 

ORGANIZATION:  U.S.  Army  Materiel  Systems  Analysis  Activity  (AMSAA) 

ORGANIZATIONAL  RESPONSIBILITY:  Performs  analysis  and  evaluation  on  major 
Army  systems.  Acts  as  an  independent  evaluator  for  DT  and  some  OT.  Uses  M&S  per¬ 
forming  analyses. 

INTERVIEW  SUMMARY:  The  interview  began  with  a  short  presentation  on  the  back¬ 
ground  and  purpose  of  the  SMART  Accreditation  Requirements  study.  During  the 
course  of  the  presentation,  several  recommendations  were  made  concerning  the 
SMART  Project  deliverables: 

-  The  CASE  tools  used  to  analyze  the  SMART  model  set  were  questioned. 
AMSAA  had  tried  to  use  CASE  tools  to  analyze  some  of  their  models  and  found 
that  they  could  not  compile  the  tools  with  their  large  FORTRAN  models. 


-  The  configuration  management  (CM)  process  to  be  developed  by  the  SMART 
project  could  be  a  concern.  In  today's  resource  constrained  environment,  a  CM 
process  which  involves  extensive  documentation  and  multiple  meetings  would 
be  too  costly.  For  models  whose  configurations  are  controlled  by  AMSAA,  one 
person  controls  changes  and  approves  distribution  of  code  to  potential  users. 
That  person  exercises  judgment  as  to  what  code  is  sent  to  a  user.  In  most 
cases  only  executable  code  is  sent. 

-  A  suggestion  was  made  that  the  model  history  to  be  recorded  in  the  SMART 
data  base  also  contain  information  on  any  comparisons  made  between  models. 

-  It  would  be  beneficial  for  SMART  to  keep  track  of  time  and  cost  estimates  for 
performing  V&V.  The  AMSAA  personnel  expressed  concern  that  the  SMART 
process  for  doing  V&V  might  be  higher  than  they  (or  anyone  else)  can  afford. 
They  presently  spend  anywhere  from  6  months  to  2  years  on  a  V&V  effort,  de¬ 
pending  on  the  size  and  extent  of  application  of  the  model.  During  that  time  they 
have  anywhere  from  1  to  3  people  working  on  the  effort.  They  felt  that  the 
SMART  efforts  will  not  negate  the  need  for  AMSAA  personnel  to  perform  their 
own  V&V  on  a  model  but  it  would  reduce  the  scope  of  the  effort  required  on  their 
part. 

-  In  discussing  SMART  contributions  to  accreditation,  a  key  factor  from  AMSAA’s 
point  of  view  was  the  value  added  to  the  accreditation  process  relative  to  the 
cost  of  production  of  accreditation  support  deliverables.  This  was  a  theme  that 
recurred  frequently  during  the  discussions. 

-  A  possible  SMART  product  suggested  by  Phil  Beavers  was  a  knowledge  based 
or  “expert  system”  piece  of  software  that  would  ask  questions  of  the  user  and 
suggest  a  V&V  outline  or  plan  depending  upon  the  level  of  accreditation  re¬ 
quired,  the  resources  available,  the  technical  requirements  of  the  accreditation 
(which  would  vary  by  both  service  and  application).  It  could  be  based  on 
AMSAA’s  V&V  checklist  as  a  starting  point.  Mr.  Beavers  said  that  such  an  ap¬ 
plication  would  provide  a  typical  user  an  “80%  solution”  to  typical  accreditation 
problems,  leaving  the  remainder  to  be  tailored  to  the  user’s  specific  situation. 

Officially,  AMSAA  accredits  models  only  for  their  own  use.  Unofficially  they  are  respon¬ 
sible  for  accrediting  models  for  which  they  have  CM  responsibility.  They  also  are  in¬ 
volved  in  accrediting  models  for  joint  use  and  for  use  in  COEAs  where  the  model  usage 
crosses  organizational  lines. 

AMSAA  has  recently  accredited  two  models  for  their  use  in  particular  applications. 
These  models  are  EVADE  and  LELAWS.  The  LELAWS  accreditation  was  a  20  month 
study  involving  1-3  people.  It  did  not  involve  any  separate  test  data  collection  effort.  All 
the  data  used  in  the  V&V  was  obtained  by  piggybacking  on  other  tests. 

In  terms  of  the  draft  Navy  VV&A  policy  AMSAA  typically  provides  a  "level  3"  accredita¬ 
tion.  The  LELAWS  accreditation  package  would  probably  meet  "level  4"  accreditation 
requirements.  No  model  has  been  approved  for  “domain  accreditation”  at  AMSAA.  A 
copy  of  a  typical  accreditation  package,  which  consisted  of  hard  copies  of  the  view 


C-2 


graphs  used  in  presenting  the  supporting  information  to  the  director  of  AMSAA,  was 
provided.  In  order  to  conserve  effort  and  paper,  annotated  copies  of  the  briefing  slides 
are  used  to  document  the  V&V  effort  vice  a  formal  V&V  report.  An  accreditation  request 
briefing  may  take  about  a  half  day  and  be  presented  to  the  director  and  various  key 
AMSAA  personnel. 

AMSAA  has  both  a  checklist  and  a  questionnaire  which  are  used  as  guidance  in 
preparing  an  accreditation  request  briefing.  Copies  of  each  of  these  were  furnished. 

AMSAA  accreditation  efforts  to  date  have  been  for  total  models.  They  have  not  per¬ 
formed  V&V  on  model  changes  in  order  to  accredit  an  updated  version  of  a  model  which 
was  previously  accredited.  If  they  were  to  accredit  an  updated  model,  they  would  prob¬ 
ably  perform  a  sensitivity  analysis  on  the  changes  and  updates  to  the  model,  and  then 
do  a  V&V  on  the  changed  portions  only.  If  some  other  activity  had  accredited  a  previ¬ 
ous  version  of  a  model  which  they  intended  to  use,  they  would  look  at  the  V&V  history 
and  use  that  as  a  basis  for  developing  an  accreditation  plan  for  the  updated  model. 

AMSAA  is  being  directed  to  provide  accreditation  support  for  7  government  and  10  con¬ 
tractor  models  in  support  of  the  RAH-66  Commanche  COEA,  although  the  time  frame 
for  completion  depends  on  budget  uncertainties.  SMART  will  send  Duke  Paris  its  most 
recent  reports  on  ESAMS  and  RADGUNS  both  for  technical  review  and  possible  later 
use  in  the  COEA  effort. 

In  preparing  an  accreditation  plan,  the  user’s  and  analyst’s  manuals  are  important. 
They  don't  feel  that  typical  software  life  cycle  documentation  is  necessary  to  support  ac¬ 
creditation.  Any  documentation  of  methodology  studies  is  important.  (Methodology 
studies  documents  are  descriptions  of  the  physics,  equations,  and  possibly  the  imple¬ 
mentation  of  a  part  of  a  model  such  as  the  P«  model  for  a  particular  gun.  A  typical 
model  part  may  be  common  to  several  models  and  described  in  a  methodology  study.) 

AMSAA  considers  "benchmarking"  (the  comparison  of  one  model's  results  with  those  of 
another  model)  another  useful  aspect  of  the  accreditation  process.  It  is  especially  use¬ 
ful  if  one  of  the  models  has  already  been  accredited. 

One  tool  that  AMSAA  uses  in  verifying  a  model  is  the  McCabe  Complexity  Rating.  It 
identifies  the  most  significant  modules  in  a  software  program  by  reading,  segregating, 
counting,  and  listing  the  number  of  logical  decisions  by  subroutine.  This  tool  is  based 
on  the  theory,  demonstrated  in  several  programs,  that  the  number  of  errors  in  a  subrou¬ 
tine  is  directly  related  to  the  number  of  decisions  which  are  made  in  the  subroutine. 
SMART  can  use  such  a  tool  to  help  in  the  prioritization  of  functional  elements  for  V&V 
efforts  on  its  models.  Mr.  Paris  agreed  that  he  could  help  us  get  this  tool. 

The  following  personnel  were  suggested  as  those  who  might  be  able  to  provide  addi¬ 
tional  information  on  accreditation  requirements: 

Roy  Willoughby,  Louis  Dominguez  TRAC/WSMR  CASTFOREM  CM 

Annette  Ratzenburger  MSMA 

Walt  Hollis  DUSA(OR)  set  up  MSMA 

COL  Knox 
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ARS  INTERVIEW  REPORT 


INTERVIEWEES:  Mr.  Greg  Gurnsey  DATE:  30  March  1993 

Ms.  Margaret  Wolchak 

ORGANIZATION:  U.S.  Army  Operational  Test  and  Evaluation  Command  (OPTEC) 

ORGANIZATIONAL  RESPONSIBILITY:  Performs  operational  tests  on  Army  systems. 
Uses  M&S  to  plan  tests  and  extrapolate  test  results,  and  to  overcome  limitations  and 
constraints  on  OT. 

INTERVIEW  SUMMARY:  The  interview  began  with  a  short  presentation  on  the  back¬ 
ground  and  purpose  of  the  SMART  Accreditation  Requirements  study.  During  the 
course  of  the  presentation,  several  issues  were  raised  which  might  impact  the  project. 
These  issues  are: 

-  The  specific  CASE  tools  used  to  analyze  the  SMART  model  set  were  ques¬ 
tioned.  OEC  questioned  if  the  CASE  tools  analysis  was  thorough,  using  a  com¬ 
plete  set  of  tools.  Different  CASE  tools  have  different  purposes. 

-  The  Data  Requirements  Dictionary  should  list  additional  factors  besides  mea¬ 
surement  accuracy.  Two  factors  which  are  important  are  precision  and  vari¬ 
ance. 

-  The  VV&A  history  of  a  model  must  include  identification  of  the  version  of  the 
model  that  was  accredited. 

-  The  term  "Unified  CM  Process"  should  be  revised  to  "Consistent  CM  Process". 
This  will  avoid  confusion  since  "Unified"  has  a  different  connotation  meaning 
multi-service  within  DOD. 

-  The  commonalty  matrix  which  identifies  common  functional  elements  in  different 
versions  of  a  model  is  now  based  on  the  similarities  between  the  functional  ele¬ 
ments.  It  should  also  consider  the  interfaces  between  the  various  Functional 
Elements  (FEs)  since  commonalty  can  be  affected  by  the  interfaces  between 
FEs. 

V&V  is  really  a  process  to  collect  evidence  to  support  a  decision  (accreditation)  that  a 
model  is  suitable  for  a  particular  application  .  The  definition  of  acceptability  criteria  is 
critical  to  making  a  valid  decision.  These  criteria  vary  with  every  application.  OEC  has 
published  a  closed  set  of  techniques  for  generating  acceptability  criteria.  These  are 
contained  in  the  OEC  IPG  92-2.  When  planning  a  V&V  effort  the  cost  of  collecting  data, 
as  well  as  other  suitability  information,  is  important.  OEC  would  benefit  if  the  SMART 
Project  could  provide  information  on  the  cost  of  collecting  different  types  of  data.  With 
regard  to  costs,  Mr.  Guernsey  suggested  that  tracking  the  cost  of  the  various  compo¬ 
nents  of  accreditation  would  also  be  value  as  an  aid  to  planning  V&V  efforts  within  a  re¬ 
source  constrained  environment. 
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Within  OEC  accreditation  is  always  driven  by  the  application.  The  MOEs  and  MOPs  for 
an  accreditation  are  based  on  the  data  elements  or  the  conditions  of  the  application.  A 
copy  of  an  accreditation  report  was  provided  to  the  SMART  Project  personnel  to  serve 
as  a  model  and  provide  information  on  OEC's  approach  to  accreditation. 

Mr.  Gurnsey  offered  the  following  observations  on  VV&A  processes  in  general: 

-  Validated  tactical  environmental  data  is  very  important  for  any  analysis. 

-  Military  judgment  is  very  important  in  doing  face  validation.  This  critical  part  of 
V&V  is  missing  in  the  SMART  Project. 

-  Any  lessons  learned  about  data  recording  or  model  validation  could  be  sent  to 
the  ranges  to  assist  them  in  structuring  future  tests. 


-  The  Army  MSMA  has  established  an  Army  Data  Dictionary  related  to  M&S. 
SMART  personnel  should  coordinate  with  them  to  help  develop  common  termi¬ 
nology  across  project  and  DOD  lines. 


-  DIS  (Distributed  Interactive  Simulation)  is  chartering  a  VV&A  working  group, 
chaired  by  Dr.  Pace,  that  will  have  a  formal  meeting  in  summer  or  fall.  There  will 
be  a  call  for  papers  shortly.  This  is  an  opportunity  for  presenting  the  SMART 
concept  to  a  wide  audience.  The  group  is  looking  to  define  a  VV&A  process  and 
will  be  looking  for  validated  data.  This  is  a  possible  overlap  area  with  the 
SMART  Project. 


Several  additional  points  of  contact  were  identified.  They  are: 


Mike  Bauman 
CW4  Weidel 


Lana  McGlynn 
Don  Blanton 

Irwin  Atzinger 


TRADOC  Analysis  Command 
Ft.  Leavenworth 
TEXCOM,  Ft  Hood 
(Commanche  simulators,  LFT  of 
Hellfire  missiles  coming  up) 

MSMA  (also  Annette  Ratzenburger) 
AMSAA  (Assist  transition  planning, 
Army  Data  Dictionary) 

AMSAA 


(913)  684-4689 
(817)  288-9992 

(703)  607-3385 
(410)  278-3368 

(410)  278-6576 
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ARS  INTERVIEW  REPORT 


INTERVIEWEE:  Dr.  Anthony  G.  Kassos  DATE:  3  February  1993 

ORGANIZATION:  U.S.  Army  Aviation  and  Troop  Command  (ATCOM) 

ORGANIZATIONAL  RESPONSIBILITY:  Dr.  Kassos  is  involved  in  system  assessments 
using  models  and  simulations.  He  will  assist  program  managers  in  accrediting  models 
for  this  analysis. 

INTERVIEW  SUMMARY:  Dr.  Kassos  provided  a  copy  of  the  new  Army  Regulation  (AR) 
5-1 1 .  He  has  not  yet  done  an  accreditation  under  these  new  guidelines.  He  is  planning 
to  start  an  accreditation  soon.  He  will  be  using  the  SMART  approach  in  this  accredita¬ 
tion.  If  any  process  problems  arise  he  will  resolve  them  as  he  sees  best. 

AR  5-1 1  applies  throughout  the  Army.  Any  previous  guidelines  by  individual  commands 
have  either  been  superseded  or  rewritten  to  conform  to  AR  5-1 1 .  Throughout  the  army, 
no  models  have  as  yet  been  accredited  for  a  "class  of  applications"  as  provided  for  in 
this  regulation. 

In  response  to  a  question  about  any  other  products  which  should  be  provided  by 
SMART,  Dr.  Kassos  indicated  that  the  results  of  a  "Face  Validation"  should  be  included 
in  the  model  validation  reports.  He  felt  that  such  a  face  validation  is  actually  done  as 
part  of  the  SMART  assessment  process. 

Within  ATCOM  they  have  not  validated  any  models  with  test  data.  Several  attempts 
have  been  made  but  none  were  very  successful.  In  general,  testing  has  been  done  to 
support  model  development. 

AR  5-1 1  makes  the  model  proponent  responsible  for  configuration  management.  The 
following  persons  have  that  responsibility  for  the  models  shown: 

RADGUNS  Dwight  FitzsimmonsFSTC 
HELIPAC  Jeff  Gavlinski  AMSAA 

Another  point  of  contact  for  possible  additional  information  on  accreditation  is  Dan 
Willard  who  is  a  member  of  the  SMART  Steering  Group. 

Dr.  Kassos  provided  information  on  the  Army  organizations  which  use  M&S.  An  organi¬ 
zation  diagram  is  attached. 
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ARS  INTERVIEW  REPORT 


INTERVIEWEES:  CAPT  Simpson  DATE:  24  May  1 993 

ORGANIZATION:  Office  of  the  Technical  Advisor,  Deputy  Chief  of  Staff  for 
Operations,  U.S.  Army 

ORGANIZATIONAL  RESPONSIBILITY:  The  office  of  the  Technical  Advisor  conducts 
studies  and  performs  analyses  to  support  headquarters  decisions.  They  are  the  accred¬ 
itation  proponents  for  COEAs. 

INTERVIEW  SUMMARY:  The  analysts  in  this  office  prepare  accreditation  information 
to  support  accreditation  of  models  used  in  COEAs.  They  have  not  yet  accredited  any 
models.  They  are  presently  involved  in  accrediting  one  model.  One  of  the  difficulties  is 
that  they  use  Force  Level  models  in  COEAs  rather  than  engineering  level  models. 
Force  level  models  are  very  difficult  to  validate  and  accredit  since  there  are  a  number  of 
human  decisions  incorporated  in  force  level  models.  Such  decisions  are  hard  to  vali¬ 
date. 

When  performing  an  analysis,  they  try  to  employ  the  "semi-formal  process".  However,  if 
time  constraints  dictate,  they  may  switch  and  use  the  "informal  process".  If  they  intend 
to  use  an  existing  model,  they  try  to  use  the  alternate  process  which  gives  the  study 
authority  the  opportunity  to  approve  or  reject  the  selection  of  the  model.  If  the  model 
requires  some  modification,  the  V&V  tends  to  be  done  in  parallel  with  the  analysis  and 
the  accreditation  is  almost  in  parallel  with  the  analytical  results. 

They  are  interested  in  Configuration  Management.  They  want  to  know  that  the  model 
they  use  is  what  they  actually  intended  to  use. 

TRADOC  at  WSMR  will  probably  do  a  COEA  on  the  Unmanned  Air  Vehicle  (UAV)  using 
CASTFORM 

CAPT  Simpson  provided  me  with  an  outline  for  a  generic  accreditation  support  data 
package.  This  outline  was  taken  from  an  in  house  paper  and  is  being  used  to  guide  the 
current  accreditation. 
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ARS  INTERVIEW  REPORT 


INTERVIEWEE:  Dr.  Dennis  Shea  DATE:  30  March  1993 

ORGANIZATION:  Center  for  Naval  Analyses  (CNA) 

ORGANIZATIONAL  RESPONSIBILITY:  Performs  studies  and  analysis,  primarily  for 
OPNAV. 

INTERVIEW  SUMMARY:  When  CNA  performs  an  analysis  using  a  legacy  model  they 
first  review  the  model  to  understand  the  approach,  algorithms,  and  assumptions  em¬ 
bedded  in  the  model.  Using  this  information,  they  decide  if  the  model  is  appropriate  for 
the  intended  application.  Although  they  have  not  been  requested  to  perform  any  code 
verification  as  part  of  their  model  reviews  for  other  Navy  activities,  they  do  perform  a 
"logical  verification"  on  such  a  model.  Logical  verification  includes  the  following  steps: 

-  evaluate  the  correctness  of  the  algorithms 

-  identify  assumptions  needed  to  use  the  model 

-  identify  variables  included/excluded  in  the  model 

-  confirm  consistency  of  units  and  coordinate  systems  throughout  the  model 

-  explore  implications  of  time-step  sizes. 

As  part  of  a  model  review,  they  do  perform  sensitivity  analyses  if  the  model  is  complete, 
documented,  and  can  be  ported  to  a  CNA  computer.  They  would  also  identify  the  limi¬ 
tations  on  the  range  of  applicability  for  the  algorithms  used  in  the  model.  If  a  more  thor¬ 
ough  VV&A  were  necessary  and  there  were  insufficient  documentation,  they  would  be¬ 
gin  with  the  code  and  reverse  engineer  the  model  to  develop  the  diagrams  needed  to  do 
a  VV&A.  They  have  performed  this  type  of  V&V  on  the  CMAS  model  as  well  as  approx¬ 
imately  30  other  models  developed  for  Navy  use.  Release  of  the  VV&A  for  report  for 
CMAS  must  be  approved  by  N81 . 

The  present  approach  relies  extensively  on  the  competence  of  the  CNA  analysts.  Dr. 
Shea  feels  that  the  present  approach  is  sufficient.  If  a  more  formal  VV&A  procedure  is 
required,  he  feels  that  it  must  be  relatively  cheap  when  compared  to  model 
development  costs.  The  present  review  takes  anywhere  from  6  months  to  1+  years  for 
a  given  model.  Since  the  Navy  has,  perhaps  hundreds  of  existing  models,  a  formal 
review  of  each  which  employs  the  existing  approach  would  be  unaffordable.  To  be 
helpful  to  a  sponsor,  a  formal  VV&A  approach  should  result  in: 

-  sufficient  evidence  to  accredit  a  model  or, 

-  recommendations  to  correct  or  work  around  noted  deficiencies  or, 

-  an  alternative  approach  to  the  pending  analytical  issue  (for  which  the  model  has 
been  proposed  as  an  analytical  tool)  that  does  not  require  the  model  under  re¬ 
view.  This  might  be  an  alternative  model  or  an  analytical  approach  that  does  not 
require  a  computer  model. 

The  final  report  should  also  describe  the  class  of  problems  for  which  the  model  can  be 
used. 

Dr.  Shea  commented  that  an  approach  to  a  detailed  VV&A  which  required  anywhere 
from  1/2  to  1+  years  time  and  resources  for  each  of  the  Navy's  existing  models  would 
probably  be  unaffordable.  Navy  sponsors  usually  can't  afford  the  time  it  would  take  to 
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carry  out  a  detailed  VV&A,  and  might  view  such  a  formal  VV&A  requirement  as  a  bu¬ 
reaucratic  burden.  Dr.  Shea  feels  that  there  is  no  opposition  in  the  Navy  to  having  a 
detailed  VV&A  procedure;  the  opposition  would  arise  when  study  sponsors  were  forced 
to  pay  for  VV&A.  Dr.  Shea  commented  that  study  sponsors  don’t  want  to  hold  up  a 
study  just  to  complete  formal  VV&A  requirements.  From  Dr.  Shea’s  point  of  view,  the 
Navy  doesn’t  think  VV&A  is  affordable  for  existing  models  and  simulations,  implying  that 
some  sort  of  “grandfather”  accreditation  for  these  models  might  be  appropriate. 

The  Navy  currently  has  no  requirement  specifying  the  level  of  V&V  required  to  accredit 
models  for  various  applications.  However,  when  N81  funds  CNA  analysis,  it  expects 
the  models  used  to  be  reviewed  using  the  support  funds. 

As  a  side  issue,  it  was  reported  that  Dale  Pace,  who  is  drafting  the  Navy  VV&A  policy, 
believes  that  CASE  tools  can  significantly  reduce  the  time  required  to  perform  VV&A  on 
models  under  development  by  automating  software  documentation.  Since  poor  docu¬ 
mentation  has  been  the  greatest  obstacle  to  performing  VV&A,  any  techniques  to  auto¬ 
mate  this  process  would  be  a  step  in  the  right  direction. 

Dr.  Shea  noted  that  the  present  process  for  releasing  and  publishing  model  changes  is 
painfully  slow.  Documentation  of  the  changes  significantly  lags  the  changes  to  the  soft¬ 
ware.  Any  C/M  system  developed  by  the  SMART  project  should  speed  up  the  process 
for  releasing  and  publishing  model  changes. 
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ARS  INTERVIEW  REPORT 


INTERVIEWEE:  CAPT  Raymond  Morris  DATE:  2  April  1993 

ORGANIZATION:  Naval  Air  Systems  Command,  Warfare  Analysis  Division  (AIR-526) 

ORGANIZATIONAL  RESPONSIBILITY:  Performs  analysis  on  proposed  weapons  and 
systems.  Uses  models  and  simulations  for  analyses  which  impact  COEAs  and  which 
are  inputs  for  advanced  planning. 

INTERVIEW  SUMMARY:  In  planning  a  study,  AIR-526  usually  assigns  a  team  consist¬ 
ing  of  a  military  officer,  a  senior  analyst,  and  a  junior  analyst.  They  prepare  a  study 
proposal  which  outlines  how  the  analysis  will  be  performed,  alternative  approaches 
which  may  be  tried,  and  candidate  models  for  the  analysis.  The  models  typically  used 
are  ones  which  have  been  designed  and  used  to  give  good  first  order  insights  and  solu¬ 
tions  to  military  problems.  Results  of  these  models  have  held  up  well  under  empirical 
scrutiny  in  past  studies. 

When  completed,  the  study  plan  is  reviewed  and  presented  to  the  study  chartering  au¬ 
thority  for  approval.  Having  received  and  approval,  AIR-526  considers  the  models 
which  were  cited  in  the  plan  as  being  acceptable  for  use  in  the  study.  This  does  not 
preclude  additional  models  from  being  employed  as  the  study  progresses. 

In  performing  the  analysis,  models  are  used  to  predict  results  at  the  1  -v-1  and  M-v-N 
levels.  Comparative  runs  are  made  with  different  models  and  the  results  are  scrutinized 
for  realism  and  consistency.  Analysts  often  combine  the  results  of  several  simple  mod¬ 
els  into  a  larger  context  through  a  variety  of  methods  including  "spreadsheet"  models  or 
Monte  Carlo  type  simulations  to  analyze  battle  force  and  campaign  level  outcomes. 
The  underlying  principle  of  these  studies  is  to  analyze  and  explain  why  a  certain  result 
occurred  and  what  parameters  drove  those  results.  AIR-526  usually  avoids  using  cam¬ 
paign  level  models,  since  the  results  are  difficult  to  break  down  into  individual  engage¬ 
ments  to  determine  the  effects  of  changes  in  the  scenario. 

Since  several  variations  in  system  design  as  well  as  the  scenario  might  be  analyzed, 
the  models  are  typically  refined  several  times  during  an  analysis  in  order  to  investigate 
the  effects  of  these  variations.  Therefore  the  models  must  be  robust,  simple  to  use,  and 
economical  to  run.  This  factor  mitigates  against  a  formal  VV&A  process  for  each  variant 
of  each  model  used  in  the  analysis. 

Because  the  studies  are  typically  of  a  short  duration,  there  is  neither  time  nor  money 
available  to  conduct  a  formal  V&V  of  each  model  and  each  variant.  The  study  director 
is  primarily  responsible  for  ensuring  that  the  appropriate  model(s)  are  used,  any 
changes  to  the  models  are  logically  correct,  and  that  these  changes  are  coded  correctly. 
Within  AIR-526  the  principle  means  of  building  confidence  in  the  analytical  results  ob¬ 
tained  through  the  use  of  models  is  through  the  use  of  multiple  models  and  by  conduct¬ 
ing  informal  peer  reviews.  Since  multiple  models  are  employed  in  a  problem,  analytical 
results  from  each  model  are  compared  with  each  other.  The  results  are  also  checked 
against  the  experience  of  senior  analysts  and  active  duty  military  fleet  operators  who 
are  part  of  the  analysis  team.  (In  essence  a  face  validation  is  performed.)  During  the 


C-11 


course  of  a  study,  informal  reviews  by  other  senior  analysts  are  conducted  to  help 
identify  any  logical  gaps  or  errors  in  the  analysis  or  model  results.  (These  reviews  are 
similar  to  "Peer  reviews"  used  by  other  organizations  in  performing  model  V&V.)  At 
each  major  briefing  of  the  study  results  to  sponsors  and/or  analysts,  an  implicit  “go- 
ahead”  is  obtained  if  the  results  square  with  experience  and  correlate  with  similar  uses 
of  the  models  in  other  parallel  studies.  Through  this  comparative  analysis  and  review 
process,  AIR-526  builds  confidence  in  the  study  results.  The  present  process  is  rela¬ 
tively  inexpensive  and  well  suited  to  operations  analysis  problems. 

Since  operations  analysis  is  most  useful  in  exploring  new  systems  concepts,  existing 
models  do  not  generally  cover  the  full  parametric  range.  Therefore,  operations  analysis 
models  tend  to  evolve  in  a  dynamic  fashion.  The  lifetime  of  a  particular  model  configu¬ 
ration  is  often  one  study.  If  a  highly  formal  VV&A  process  were  forced  upon  AIR-526, 
these  COEA  studies  would  be  greatly  stretched  out  and  the  study  costs  would  rise  sig¬ 
nificantly.  VV&A  is  more  appropriate  for  models  which  are  used  repetitively  (e.g.  train¬ 
ing  models,  simulators,  etc.)  The  COEA  process  should  not  be  burdened  with  a  re¬ 
quirement  for  a  formal  VV&A  of  each  model.  Flexibility  in  the  early  phases  of  analysis  is 
essential  to  producing  timely  results.  A  strict  VV&A  policy  for  M&S  would  tie  the  hands 
of  analysis  agencies  too  tightly  to  be  effective. 
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ARS  INTERVIEW  REPORT 


INTERVIEWEES:  Mr.  James  Duff  (Technical  Director)  DATE:  29  March  1993 
CDR  Barry  Kelly 

ORGANIZATION:  Operational  Test  and  Evaluation  Force  (OPTEVFOR) 

ORGANIZATIONAL  RESPONSIBILITY:  Responsible  for  performing  operational  tests, 
evaluating  new  systems,  and  determining  their  suitability  for  Navy  use. 

INTERVIEW  SUMMARY:  OPTEVFOR  established  an  analysis  group  only  two  years 
ago.  The  group  has  grown  from  a  single  M&S  “expert”  to  a  division  with  12-15  ana¬ 
lyst/modelers  and  one  software  “expert,”  distributed  among  three  branches:  Simulation 
and  Modeling,  Analysis,  and  Software.  CDR  Kelly  heads  the  Simulation  and  Modeling 
branch.  Due  to  the  cost  and  security  issues  surrounding  testing,  simulation  and  model¬ 
ing  is  becoming  more  widely  used  in  OPEVAL. 

Current  OPTEVFOR  policy  guidance  on  the  use  of  M&S  in  OT  discourages  the  devel¬ 
opment  of  new  M&S,  but  use  of  existing  M&S  is  acceptable.  The  OPTEVFOR  focus  is 
on  the  development  and  enhancement  of  expertise  in  adapting  and  using  models  for  OT 
purposes.  V&V  efforts  in  support  of  accreditation  decisions  are  typically  contracted  out. 
The  advent  of  J-MASS  will  require  that  OPTEVFOR  develop  the  capability  to  create 
models  using  the  J-MASS  architecture.  Therefore,  OPTEVFOR  must  start  to  work  to¬ 
ward  developing  that  capability  now.  If  it  doesn't,  it  will  eventually  lose  the  capability  to 
use  M&S  effectively  to  support  future  OT.  OPTEVFOR  is  also  looking  to  get  involved 
with  the  DIS  (Distributed  Interactive  Simulation)  effort. 

OPTEVFOR  uses  models  to: 

1  Assess  the  lethality  of  weapons 

2  Extend  test  results  to  conditions  difficult  to  test 

3  Investigate  aspects  where  tests  cannot  be  performed 

4  Refine  test  planning 

5  Refine  resource  requirements 

6  Assist  in  early  operational  assessments 

7  Avoid  testing  where  safety  or  environmental  risks  are  too  high 

8  Assess  nuclear  effects 

9  Evaluate  communications  networks 

In  selecting  a  model  for  a  particular  application,  OPTEVFOR  seeks  information  from  the 
J-8  catalogue,  SURVIAC,  the  system  program  office,  and  other  generic  sources  (e.g. 
CNA,  Naval  Post  Graduate  School  [NPGS],  Navy  labs).  The  following  questions  are 
asked  when  looking  for  a  model: 

10  What  operational  questions  must  be  answered? 

1 1  Is  use  of  a  model(s)  appropriate  to  answer  these  questions? 

12  What  is  the  model  expected  to  provide  relative  to  these  questions? 

13  What  models  are  available? 


C-1 3 


14  What  is  known  about  the  model’s  intent,  assumptions,  performance  and  fi¬ 
delity?  (High  previous  usage  of  a  given  M&S  is  not  essential.) 

15  Is  the  (or  are  these)  model(s)  appropriate?  (Discuss  the  model  with  the  de¬ 
veloper  and  past  users  to  determine  its  appropriateness.) 

16  Can  the  model  be  modified  (or  a  new  model  developed)  to  suit  the  purpose? 

17  Will  the  system  program  office  support  the  development  or  modification  of 
the  selected  model? 

18  Is  there  an  independent  group  or  activity  which  can  V&V  the  model? 

19  What  is  the  V&V  history  of  the  model?  (This  is  not  an  overriding  factor.) 

OPTEVFOR  has  not  approved  the  accreditation  of  a  model  in  the  past  year,  although 
two  models  have  been  subjected  to  the  accreditation  process.  In  both  cases  the  rec¬ 
ommendation  was  for  non-accreditation.  In  the  past,  OPTEVFOR  has  accredited  the 
SIMVAL  model  for  the  SSN-21.  This  accreditation  was  done  by  CNA.  Mr.  Duff  com¬ 
mented  that  OPTEVFOR  makes  use  of  CNA  in  this  role  as  often  as  possible.  Also  BDM 
Inc.  conducted  an  accreditation  study  on  the  IADS  model  2-3  years  ago.  OPTEVFOR 
generally  has  another  activity,  independent  from  the  model  developer,  perform  the  V&V 
necessary  to  support  an  accreditation  decision.  They  feel  that  CNA  has  the  credibility  to 
do  adequate  V&V  for  their  accreditation's.  OPTEVFOR  looks  to  CNA’s  V&V  activity  to 
state  where  the  model  will  be  adequate  and  where  it  fails  to  adequately  represent  the 
conditions  of  the  application  for  which  it  is  intended  to  be  used.  The  accreditation  pro¬ 
cess  takes  anywhere  from  6  months  to  2-3  years. 

Mr.  Duff  feels  that  model  verification  is  feasible.  Model  validation,  however,  is  more  dif¬ 
ficult  and  cannot  always  be  done.  The  time  frame  for  the  use  of  the  model  can  signifi¬ 
cantly  impact  the  amount  of  V&V  which  is  possible.  Moreover,  Mr.  Duff  feels  that  we 
will  never  have  a  full  validation  of  any  model,  but  that  statements  about  the  boundaries 
of  model  applicability  are  sufficient  to  support  accreditation  decisions.  CDR  Kelly  ex¬ 
pressed  the  opinion  that  VV&A  is  a  “random  space”  that  the  services  are  now  trying  to 
put  some  order  into. 

If  a  model  is  to  be  used  to  support  an  OT  decision,  either  fully  or  partially,  the  model 
must  have  an  independent  agent  perform  V&V  in  support  of  accreditation.  The  model 
developer  can  participate,  but  must  not  be  solely  responsible  for,  this  V&V.  If  the  model 
is  to  be  used  in  test  planning,  or  in  some  other  activity  in  support  of  OT,  it  need  not  be 
accredited.  If  it  will  be  used  to  resolve  a  critical  operational  issue  (COI),  it  must  be  ac¬ 
credited. 

OPTEVFOR  has  no  immediate  need  for  the  type  of  accreditation  support  material  pro¬ 
vided  by  SMART  for  the  SMART  model  set.  However,  SMART’S  general  accreditation 
support  products  (V&V  Process  Guides,  Configuration  Management  Handbook,  VV&A 
Data  Base,  etc.)  would  provide  a  useful  “stake  in  the  ground”  for  V&V  and  C/M  activities 
in  support  of  accreditation. 

In  response  to  a  question  as  to  whether  survivability  models  are  used  by  OPTEVFOR, 
CDR  Kelly  indicated  that  the  use  of  survivability  models  would  require  the  development 
of  a  system  model  to  interface  with  the  survivability  model.  The  cost  of  this  develop¬ 
ment  would  be  a  problem.  It  would  probably  be  less  expensive  to  actually  exercise  the 
system  at  Echo  range  than  to  develop  the  necessary  models  to  interface  with  existing 
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survivability  models.  CDR  Kelly  indicated  that  most  survivability  COIs  are  resolved  by 
the  time  the  system  gets  to  OT.  If  survivability  is  still  a  COI  in  OT,  the  system  program 
manager  will  probably  describe  what  was  done  in  DT  and  suggest  an  approach  to  be 
used  in  OT.  OPTEVFOR  will  either  accept  or  modify  that  approach  to  make  a  judgment 
regarding  the  COI.  OPTEVFOR  typically  uses  system-specific  models  rather  than 
generic  models  to  support  OT  evaluations. 

Mr.  Duff  mentioned  his  concept  of  a  "Statement  of  Functionality."  If  tactical  software  is 
modified,  a  statement  of  functionality  should  be  developed  to  identify  the  functional  ef¬ 
fect  and  limitations  of  that  particular  modification.  If  a  model  is  changed  and  must  be 
reaccredited,  a  statement  of  functionality  about  the  model  changes  would  be  very  useful 
in  guiding  the  additional  V&V  which  should  be  done  to  accredit  the  changed  model. 

The  following  personnel  were  suggested  as  possible  additional  contacts: 


Name 

Role 

Phone 

Jim  Sikora  (BDM, 
Albuquerque) 

Prepared  MORS  accreditation  guide. 

Will  publish  a  white  paper  on  the  ac¬ 
creditation  process.  Knows  attendees 
(especially  service  reps)  at  meeting 
where  templates  were  discussed. 

4104  848-5650 

Dee  Ritchie 
(DSMC,  Ft.  Belvoir) 

Knowledgeable  on  accreditation  re¬ 
quirements.  Editor  of  PFIALANX 
(MORS  newsletter).  Knowledgeable  on 
MORS  SIMVAL. 

4105  805-2887  or 
(703)  805-3698 

Pat  Sanders,  PA&E 

4106  697-3521 

Steve  Booher 
(Teledyne  Brown 
Engineering) 

Developed  model  similar  to  ESAMS. 

Can  discuss  V&V  on  that  model. 

4107  276-4603 
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ARS  INTERVIEW  REPORT 


INTERVIEWEE:  Mr.  Jim  Weatherly  DATE:  30  March  1993 

Mr.  Gary  L.  Keck 

ORGANIZATION:  Space  and  Naval  Warfare  Systems  Command  (SPA WAR  31) 

ORGANIZATIONAL  RESPONSIBILITY:  Tasked  by  OPNAV  (N81)  to  draft  Navy  policy 
for  conducting  and  managing  VV&A  on  models  and  simulations. 

INTERVIEW  SUMMARY:  SPAWAR  is  leading  the  effort  to  develop  a  Navy  policy  for 
VV&A  of  M&S.  Mr.  Weatherly  felt  that  the  SMART  process,  as  outlined  in  the  presenta¬ 
tion  given  by  Paul  Muessig,  would  form  the  basis  for  performing  VV&A  on  legacy  mod¬ 
els.  The  draft  policy  primarily  addresses  VV&A  for  newly  developed  models. 

In  order  to  accredit  legacy  models  under  the  proposed  policy,  he  thought  that  the 
OPNAV  sponsor  would  be  the  accrediting  authority  and  the  V&V  proponent  would  sub¬ 
mit  the  information  to  support  an  accreditation  decision.  The  amount  of  V&V  would  de¬ 
pend  on  the  application,  just  as  with  new  models.  For  example,  if  safety  of  flight  is  in¬ 
volved,  a  greater  amount  of  V&V  would  be  required  than  if  the  model  were  to  be  used 
only  in  test  planning. 

It  was  suggested  that  the  SMART  approach  to  VV&A  be  briefed  at  the  summer  meeting 
of  Team  Mike,  which  is  the  advisory  group  that  guides  the  development  of  the  Navy's 
VV&A  policy.  The  briefing  should  compare  our  outputs  and  processes  with  those  out¬ 
lined  in  the  draft  Navy  policy  document.  It  should  also  point  out  those  things  which 
SMART  is  doing  which  are  not  covered  in  the  Navy  document.  In  essence,  the  SMART 
approach  may  serve  as  a  “validation”  of  the  Navy’s  proposed  VV&A  paradigm  for  the 
important  class  of  legacy  models.  The  SMART  approach  may  also  help  the  refine  the 
policy  on  non-Navy  models  used  in  support  of  Navy  studies. 

One  approach  for  generating  this  briefing  is  to  write  an  accreditation  plan  for  a  model, 
turn  the  process  into  a  flow  chart,  and  compare  this  flow  chart  with  that  in  the  draft  Navy 
policy  document. 

Dr.  Dale  Pace  at  the  Johns  Hopkins  University  Applied  Physics  Laboratory  (JHU/APL) 
should  be  contacted  so  that  the  briefing  can  be  coordinated  with  any  changes  or  modifi¬ 
cations  which  might  be  made  to  the  draft  Navy  policy.  A  strawman  outline  of  the  issues 
to  be  briefed  should  be  provided  to  Weatherly  in  about  6  weeks  to  2  months. 
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ARS  INTERVIEW  REPORT 


INTERVIEWEE:  Dr.  Dale  Pace  DATE:  25  May  1993 

Ms.  Simone  Youngblood 

ORGANIZATION:  Johns  Hopkins  University,  Applied  Physics  Laboratory 

ORGANIZATIONAL  RESPONSIBILITY:  Primary  contractor  responsible  for  developing 
the  Navy  policies  for  M&S  VV&A.  Drafted  the  existing  Interim  Policy  Guidance  for  the 
Navy. 

INTERVIEW  SUMMARY:  Dr.  Pace  indicated  that  the  concepts  contained  in  the  draft 
VV&A  IPG  is  based  on  his  work  in  the  field  of  model  VV&A.  One  major  effort  was  his 
task  to  perform  an  independent  validation  review  of  several  models  for  SPAWAR  31. 
Dr.  Pace  provided  a  disk  with  a  bibliography  list  which  he  used  in  preparing  the  draft 
Navy  IPG. 

During  the  course  of  this  and  other  meetings  on  the  IPG,  Dr.  Pace  stated  that  the  most 
important  part  of  the  V&V  process  which  is  outlined  in  the  IPG  is  the  Concept  Validation 
Phase.  The  conceptual  validation  includes  a  check  of  assumptions,  limitations,  and  de¬ 
scription  of  the  subject  being  modeled.  (These  factors  have  been  cited  by  several  other 
interviewees  as  being  important.)  He  also  expressed  the  view  that  M&S  are  end  prod¬ 
ucts  in  themselves.  In  that  respect  they  should  have  a  structured  VV&A  process  which 
each  must  undergo  to  ensure  that  they  are  as  good  as  possible  when  released  to  the 
user  community. 

In  response  to  a  question  about  the  feasibility  of  assembling  a  team  of  independent  ex¬ 
perts  to  conduct  each  phase  of  the  V&V  effort  outlined  in  the  IPG,  Dr.  Pace  indicated 
that  the  team  need  not  be  large.  In  fact,  for  some  simple  models,  the  team  may  be  only 
one  person  to  review  what  the  model  developer  has  done.  The  independent  person 
may  even  be  from  the  same  activity  as  the  model  developer  in  order  to  overcome  the 
problem  of  finding  available  people. 
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ARS  INTERVIEW  REPORT 


INTERVIEWEE:  Ms.  Gabriella  Russell  DATE:  30  March  1993 

ORGANIZATION:  Tomahawk  Cruise  Missile  Project  Office  (PMA-281) 

ORGANIZATIONAL  RESPONSIBILITY:  Responsible  for  using  various  models  to  gen¬ 
erate  data  which  are  used  in  the  Theater  Mission  Planning  System  (TMPS).  Uses  sur¬ 
vivability  models  to  generate  Pr  data  for  the  TMPS. 

INTERVIEW  SUMMARY:  The  Tomahawk  Program  is  engaged  in  a  major  upgrade  to 
the  TMPS.  The  title  for  this  effort  is  the  Theater  Mission  Planning  Center  Upgrade 
(TMPCU).  Testing  of  this  system  will  be  done  by  OPTEVFOR  prior  to  its  certification  for 
operational  use. 

Initially,  STRATCOM  provided  data  which  was  generated  from  ROPES  to  input  to  the 
TMPS.  During  Desert  Storm  the  Navy  found  that  the  STRATCOM  data  didn't  fit  the 
TMPS  planning  needs.  Now  the  project  office  is  using  ESAMS,  RADGUNS,  TRAP,  and 
AASPEM  to  generate  data  for  the  TMPCU.  These  models  were  selected  at  the  direc¬ 
tion  of  VAdm  Bowes.  One  of  the  concerns  in  selecting  these  models  was  that  none  of 
them  were  proprietary.  Ms.  Russell  indicated  that  accreditation  of  these  models  by 
some  outside  agency  would  be  very  beneficial,  but  that  her  group  has  no  plans  (and  no 
funds)  to  do  so. 

The  Tomahawk  Project  Office  does  not  have  a  formalized  V&V  process  for  these  (or 
any  other  models)  which  are  used  in  conjunction  with  the  TMPS  or  TMPCU.  The 
Applied  Physics  Lab  at  Johns  Hopkins  University  (JHU/APL)  has  “validated”  other 
models,  but  the  survivability  models  have  not  been  validated  or  formally  accredited.  Ms. 
Russell’s  group  runs  the  models  and  compares  the  results  with  test  data  which  have 
been  collected.  If  anomalies  in  the  models  are  detected,  they  are  reported  to 
SURVIAC.  Sometimes,  McDonnell  Douglas  is  requested  to  analyze  the  anomalies  and 
suggest  model  changes  to  SURVIAC.  (A  typical  anomaly  might  be  that  the  Pk  obtained 
from  test  data  differs  from  the  model  predicted  value.)  Mr.  Nick  Talarico,  phone  (314) 
234-8870,  is  the  point  of  contact  at  McDonnell  Douglas  for  information  about  this  model 
analysis. 

Ms.  Russell  believes  that  the  models  should  receive  some  sort  of  a  "stamp  of  approval" 
from  some  external  source,  preferably  from  DOD.  In  particular,  some  sort  of  “domain 
accreditation”  of  M&S  for  a  range  of  applications  would  be  useful.  Currently  the 
PEO(CU)  implicitly  accredits  a  model  by  accepting  the  model  results.  The  Tomahawk 
office  does  do  an  informal  "face  validation"  on  the  input  data  which  are  used  in  the 
models. 

If  the  project  office  is  required  to  do  V&V  on  these  models,  Ms.  Russell  feels  that  code 
verification  and  comparisons  between  model  and  test  data  are  the  two  most  important 
steps.  In  comparisons  with  test  data,  the  usual  guide  for  acceptability  of  model  results 
is  that  they  fall  within  ±10%  of  the  test  data.  With  regard  to  the  utility  of  SMART  prod¬ 
ucts,  she  found  them  all  to  be  of  value  but  indicated  that  no  one  was  demanding  that 
such  documentation  be  produced  to  support  M&S  usage. 
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Configuration  control  would  also  be  a  potential  problem  since  the  user  community 
makes  changes  to  various  models.  Ms.  Russell  commented  that  users  put  changes  and 
fixes  into  the  current  C/M  system  without  regard  for  (and  supporting  analysis  of)  how 
these  changes  may  affect  the  overall  model. 

Ms.  Russell  indicated  that  she  could  run  the  models  she  is  working  with  and  provide  as¬ 
sessments  using  the  SMART  process  and  the  Tomahawk  data  if  she  were  provided  the 
necessary  funding.  The  report  from  such  an  effort  would  be  that  the  model(s)  were  or 
were  not  acceptable  for  use  in  certain  new  technology  applications. 
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ARS  INTERVIEW  REPORT 


INTERVIEWEES:  Clayton  J.  Thomas  (Chief  Scientist)  DATE:  2  April  1 993 

David  E.  Anderson  (Scientific  and  Technical  Advisor) 

LTCOL  Henry  Pugh  (Chief  Analyst) 

ORGANIZATION:  Air  Force  Studies  and  Analysis  Agency  (AFSAA) 

ORGANIZATIONAL  RESPONSIBILITY:  AFSAA  conducts  studies  and  performs  analy¬ 
ses  on  major  Air  Force  systems  and/or  operations  for  the  Secretary  of  the  Air  Force  and 
the  headquarters  offices. 

INTERVIEW  SUMMARY:  AFSAA  currently  does  not  have  a  written  policy  which  ad¬ 
dresses  VV&A  for  models  used  for  their  studies.  AFSAA  had  a  senior  review  panel 
(SRP)  which  conducted  three  reviews  of  a  study  project.  During  the  first  review,  the 
justification  for  selection  of  a  particular  model  was  reviewed.  That  process  was  sup¬ 
planted  with  an  Analysis  Review  Panel,  which  is  also  defunct.  Their  current  de  facto 
policy  is  to  use  competent  analysts  and  rely  on  them  to  scrub  the  models  they  use,  to 
ensure  that  they  are  appropriate  for  the  study  or  analysis  and  adequately  reflect  reality 
for  the  application  at  hand.  AFSAA  recently  completed  a  detailed  comparison  of  the 
STRAPEM  and  ROPES  models  that  determined  their  limits  of  applicability  to  AFSAA 
problems,  but  it  was  not  stated  whether  any  formal  process  was  used  to  complete  the 
study. 

Mr.  Thomas  believes  that  AFSAA  analysts  have  the  expertise  to  know  how  the  models 
can  be  best  used.  He  also  recognizes  that  time  constraints  may  dictate  that  some 
model  which  is  not  totally  suited  to  a  particular  problem  be  used  as  is. 

AFSAA  is  not  the  prime  activity  on  any  Air  Force  weapon  system  Cost  and  Operational 
Effectiveness  Analysis  (COEA).  Some  of  their  analyses  are  used  in  support  of  COEAs 
however.  At  times  the  use  of  a  particular  model  is  directed  from  above,  and  the  ana¬ 
lysts  must  make  the  necessary  changes  to  fit  the  model  to  the  study  scenario.  An  ex¬ 
ample  of  this  situation  was  the  use  of  TAC  BRAWLER  for  the  F-22  COEA.  In  this  case, 
TAC  BRAWLER  was  declared  to  be  the  “accredited”  model,  and  information  supporting 
this  accreditation  was  to  be  developed  during  the  COEA. 

Mr.  Thomas  believes  that  the  Air  Force  should  draft  a  centralized  VV&A  policy,  and  that 
it  should  be  as  permissive  as  the  traffic  can  bear  under  the  current  DOD  guidelines.  He 
said  he  is  working  on  such  a  policy  now  for  GEN  Mclnerny’s  signature,  but  he  did  not 
give  the  time  frame  for  completion. 

M&S  are  constantly  undergoing  change.  Mr.  Thomas  believes  that  the  focus  of  VV&A 
should  be  on  the  changes  for  models  which  have  been  in  use  for  several  years,  such 
as  ESAMS.  He  stated  that  PA&E  (Program  Analysis  and  Evaluation,  an  OSD  office) 
doesn't  have  much  impact  on  model  selection  or  accreditation  for  AFSAA  uses. 

Mr.  Thomas  perceives  the  SMART  process  as  an  scientific/engineering  approach  to 
VV&A  which  can  provide  information  to  support  accreditation  of  the  SMART  models  for 
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a  class  of  applications.  The  analyst’s  job  is  to  build  on  this  work  and  do  only  the  neces¬ 
sary  additional  work  to  ensure  that  the  model  is  adequate  for  the  particular  application. 

Mr.  Anderson  alerted  us  to  the  recent  DOD  IG  report  (93-060)  entitled 
"Duplication/Proliferation  of  Weapon's  Systems  Modeling  and  Simulation  within  DOD." 
A  copy  of  the  report  was  requested  from  the  Pentagon  publications  office. 

A  follow-on  meeting  was  suggested  to  interview  Dr.  Al  Murashige,  MAJ  William 
Behymer,  and  LTCOL.  Henry  Pugh  to  collect  more  detailed,  working  level  information. 
(LTCOL  Pugh  was  called  away  from  this  meeting  shortly  after  it  started.) 
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ARS  INTERVIEW  REPORT 


INTERVIEWEE:  Dr.  Marion  Williams  (Chief  Scientist)  DATE:  5  February  1993 

ORGANIZATION:  Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC) 

ORGANIZATIONAL  RESPONSIBILITY:  Provides  overall  technical  direction  for  the 
conduct  of  operational  tests  of  Air  Force  aeronautical  systems. 

INTERVIEW  SUMMARY:  This  interview  was  conducted  after  that  of  LTCOL  Koozin. 
The  questions  were  directed  at  specific  issues  or  areas  on  which  Dr.  Williams  could 
provide  unique  insights. 

In  response  to  a  question  about  the  criteria  used  to  evaluate  the  adequacy  of  a  M&S 
accreditation  plan,  Dr.  Williams  indicated  that  he  would  look  for  an  accurate  picture  of 
the  model.  This  "picture"  should  show  the  pieces  of  the  model  and  state  which  pieces 
would  be  compared  with  what  test  data.  The  accreditation  plan  should  also  show  the 
envelope  within  which  the  above  comparisons  are  valid,  what  pieces  of  the  model  would 
not  be  validated,  and  what  impact  or  risks  would  accrue  from  not  validating  the  indicated 
pieces. 

In  AFOTEC,  model  analytical  results  are  used  in  test  planning,  and  to  understand  the 
impact  of  conditions  that  cannot  be  tested  on  conclusions  drawn  from  Operational  Test 
and  Evaluation  (OT&E).  Due  to  Title  10  limitations  on  the  use  of  M&S  in  OT&E,  simula¬ 
tions  cannot  be  used  in  lieu  of  operational  testing  to  draw  conclusions  on  system  opera¬ 
tional  effectiveness  and  suitability.  Typically,  a  test  is  run,  the  test  results  are  compared 
with  model  predictions  to  calibrate  the  model,  the  results  of  a  test  under  different  condi¬ 
tions  are  predicted,  and  then  tests  at  the  new  test  point  are  conducted.  If  the  model  re¬ 
sults  compare  favorably  with  the  test  results  at  this  second  point,  the  model  is  consid¬ 
ered  validated  for  the  purpose  at  hand. 

In  developing  the  accreditation  process,  Dr.  Williams  was  a  key  individual  who  insisted 
that  the  user  is  responsible  for  accreditation.  He  is  rethinking  that  position.  He  believes 
that  there  may  be  some  utility  and  benefit  in  having  a  model  undergo  some  form  of 
"general  accreditation"  similar  to  that  described  in  Army  Regulation  5-11.  This  general 
accreditation  would  serve  as  a  starting  point  for  a  user's  application-specific  accredita¬ 
tion.  In  some  cases  the  general  accreditation  would  be  sufficient  to  accredit  a  model  for 
a  particular  application. 

Dr.  Williams,  along  with  a  team  of  other  service  representatives,  will  be  doing  an  inde¬ 
pendent  review  of  the  SEAS  (sp.)  model  accreditation. 

The  following  personnel  were  stated  to  be  important  players  in  M&S  accreditation: 


GEN  Mclnerny 
MGEN  Leaf  (Ret.) 
LTCOL  Bruce  Smith 
Greg  Gurnsey 


Ass't  Vice  Chief  of  Staff  Special  Ass't  for  M&S 
Heads  AF  T&E 

AFSAA  Air  Force  DMSO  representative 

OPTEC  Good  knowledge  of  accreditation 
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ARS  INTERVIEW  REPORT 


INTERVIEWEE:  LTCOL  Walter  Koozin  DATE:  3  February  1 993 

ORGANIZATION:  Air  Force  Operational  Test  and  Evaluation  Center,  Systems 
Analysis  Division  (AFOTEC/SA) 

ORGANIZATIONAL  RESPONSIBILITY:  Provides  analytical  support  to  AFOTEC  in 
evaluating  weapons'  effectiveness  during  operational  test  and  evaluation.  Also  provides 
modeling  support  to  assist  in  test  planning. 

INTERVIEW  SUMMARY:  AFOTEC  is  developing  internal  policy  guidelines  which  gov¬ 
ern  accreditation  of  M&S  used  in  support  of  OT&E.  This  policy  document  has  under¬ 
gone  several  revisions.  The  current  draft  titled  "The  AFOTEC  Accreditation  Handbook" 
dated  July  1992  (a  copy  was  provided)  is  similar  in  policy  to  the  previous  drafts.  The 
difference  is  in  the  amount  of  detailed  guidance  which  is  contained  in  the  document. 
Although  the  current  document  doesn't  address  any  independent  review  of  the  accredi¬ 
tation  support  material,  Koozin  reported  that  Dr.  Marion  Williams  was  interested  in  hav¬ 
ing  an  Independent  Review  Group  participate  in  the  accreditation  process. 

Current  AFOTEC  policy  is  that  if  analytical  results  obtained  through  the  use  of  M&S  is  to 
be  used  in  any  formal  report  to  support  an  acquisition  decision,  or  in  any  operational  as¬ 
sessment  of  a  system,  then  the  M&S  used  must  be  formally  accredited.  If  a  M&S  is  to 
be  used  merely  to  support  test  planning,  it  does  not  need  to  be  accredited.  However, 
the  SA  division  attempts  to  use  M&S  with  which  they  are  familiar  and  in  which  they  have 
some  confidence  that  the  results  are  credible. 

In  discussing  policies  of  other  AF  commands,  LTCOL  Koozin  did  not  know  what  their 
accreditation  requirements  were.  However,  his  personal  opinion  is  that  any  M&S  which 
is  used  in  tactics  development,  or  other  similar  applications  where  major  impacts  can  be 
felt,  should  be  formally  accredited. 

The  SA  division  has  not  developed  any  accreditation  plans  nor  accredited  any  models 
under  the  new  policy  guidelines  now  being  finalized.  Previously  developed 
accreditation  plans  were  not  very  good  and  should  not  be  used  as  a  model  of  a  "good" 
plan. 

Within  AFOTEC,  the  ST  (Special  Test)  division  does  configuration  management  (CM) 
for  their  changes.  They  feed  information  on  these  changes  back  to  the  user  community. 
Their  documentation  for  these  changes  may  not  be  considered  adequate.  LTCOL 
Koozin  feels  that  CM  is  a  critical  part  of  the  accreditation  decision,  especially  if  historical 
data  are  to  be  used  for  any  part  of  the  accreditation  decision. 

In  making  the  selection  of  a  model  for  a  particular  analytical  application,  the  candidate 
model  V&V  histories  should  be  reviewed.  The  information  on  past  confidence  en¬ 
hancements  should  be  part  of  the  considerations  affecting  model  selection.  It  was 
LTCOL  Koozin's  opinion  that  SMART  should  develop  its  own  assessment  of  the 
confidence  that  can  be  placed  in  each  model.  SMART  should  develop  its  own  level  of 
confidence  in  the  model  set  included  in  the  project,  and  make  specific  statements  about 
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the  confidence  that  can  be  placed  in  the  model  results.  Any  parts  of  the  models  which 
have  not  been  assessed  should  be  identified.  Those  areas  which  have  been  shown 
through  sensitivity  analyses  to  have  the  greatest  impact  on  the  output  should  also  be 
highlighted.  Such  a  general  assessment  from  SMART  would  be  useful  as  a  basis  from 
which  application  specific  assessment  may  be  performed  by  the  user. 

Currently  the  Air  Staff  (XORN)  has  “pseudo-authority”  for  M&S.  They  are  the  de  facto 
focal  point  of  contact  for  M&S  issues.  Air  Force  attention  on  M&S  accreditation  is  driven 
by  the  draft  DOD  directive  on  M&S.  A  possible  point  of  contact  for  additional  accredita¬ 
tion  discussions  is  LTCOL  Bruce  Smith  at  the  Air  Force  Studies  and  Analysis  Agency 
(AFSAA).  A  second  potential  contact  is  Rich  Woodard,  AFDTC/XRI,  at  (904)  882-3890. 

LTCOL  Koozin's  personal  opinion  is  that  models  which  have  been  developed  IAW  MIL- 
STD  2167A  do  not  require  additional  verification  as  long  as  no  changes  are  made  to  the 
model. 
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ARS  INTERVIEW  REPORT 


INTERVIEWEE:  Donald  Giadrosich  DATE:  1  October  1993 

ORGANIZATION:  Operations  Analysis  Division,  Air  Warfare  Center,  Eglin  AFB 

ORGANIZATIONAL  RESPONSIBILITY:  The  Operations  Analysis  Division  participates 
in  developing  models  used  in  trainers  and  for  training  systems.  They  also  help  the  Air 
Combat  Command  in  performing  studies  related  to  requirements  definition  and  opera¬ 
tional  demonstrations.  They  participate  in  follow-on  OT&E,  tactics  development,  and 
system  usage  planning. 

INTERVIEW  SUMMARY:  In  the  opinion  of  Mr.  Giadrosich  the  most  critical  part  of  any 
study  or  analysis  is  the  proper  selection  of  the  measures  of  performance  or  measures  of 
effectiveness  which  will  be  used  to  develop  the  study  conclusions.  Using  inappropriate 
MOPs  or  MOEs  can  lead  to  erroneous  results,  even  if  the  model  or  simulation  used  in 
the  study  is  an  accurate  representation  of  the  system(s)  under  study.  The  example  Mr. 
Giadrosich  used  was  a  study  of  two  alternate  Electronic  Counter  Measures  (ECM)  sys¬ 
tems.  A  measure  of  effectiveness  for  the  systems  might  be  the  survivability  of  the  pro¬ 
tected  aircraft.  If  the  model  produced  results  showing  that  the  aircraft  survivability  was 
equally  good  for  each  system,  the  conclusion  would  be  that  either  system  could  be 
used.  However,  if  the  aircraft  was  an  attack  aircraft  and  the  measure  of  effectiveness 
was  the  ability  of  the  aircraft  to  destroy  a  target  with  the  minimum  chance  of  aircraft 
damage,  one  system  might  prove  more  effective  than  another.  This  change  in  study 
conclusions  would  result  if  one  system  required  tactics  which  impeded  or  constrained 
the  aircraft’s  ability  to  locate  and  attack  a  target  whereas  the  other  system  imposed  no 
such  constraints. 

Another  issue  which  Mr.  Giadrosich  raised  was  the  tendency  of  analysts  and  persons 
responsible  for  directing  a  study  to  use  available  models  for  that  study  even  though  the 
selected  model  might  not  be  totally  appropriate.  The  typical  reasons  that  this  tendency 
exists  are: 

no  other  appropriate  models  exist  and  time  does  not  permit  new  model  devel¬ 
opment; 

unfamiliarity  with  other  models 

the  selected  model  is  recognized  and  accepted  by  decision  makers 
direction  of  senior  decision  makers. 
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ARS  INTERVIEW  REPORT 


INTERVIEWEES:  Richard  Woodard  DATE:  30  September  1993 

LTCOL  Randy  Enright 
Brad  Atherton 
Dr.  Choltia  Posey 
Robert  Jones 

ORGANIZATION:  Air  Force  Development  Test  Center  (AFDTC),  Eglin  AFB 

ORGANIZATIONAL  RESPONSIBILITY:  The  AFDTC  is  responsible  for  performing  de¬ 
velopment  tests  on  new  and  modified  aircraft  and  installed  systems.  They  use  models, 
particularly  missile  flyout  models,  as  part  of  their  hardware-in-the-loop  simulations. 

INTERVIEW  SUMMARY:  Mr.  Jones  and  Dr.  Posey  are  primarily  involved  in  developing 
the  models  used  in  the  various  simulations.  Mr.  Jones  was  the  principal  spokesperson 
for  the  group. 

In  order  to  provide  an  understanding  of  how  AFDTC  fits  in  with  the  other  air  force  orga¬ 
nizations,  Mr.  Jones  provided  a  partial  organizational  chart  showing  the  various  com¬ 
mands  and  their  relationships.  That  partial  chart  is  attached. 

AFDTC  does  not  have  any  formal  accreditation  requirements  for  the  models  they  de¬ 
velop  and  use.  There  is  no  effort  underway  to  develop  such  requirements.  They  do 
follow  the  SIMVAL  process  for  model  V&V  thus  ensuring  that  the  models  conform  to  the 
Defense  Intelligence  Agency  (DIA)  baselines. 

The  models  they  develop  are  typically  detailed  six  degree  of  freedom  aerodynamic 
models  of  threat  missiles  which  are  integrated  with  real  hardware  to  provide  an  overall 
simulation  of  a  threat  system.  Because  of  this  integration  with  real  hardware,  their 
models  should  have  high  fidelity  and  capable  of  real  time  or  near  real  time  operation. 
The  fidelity  requirements  vary.  Typically  their  design  goal  is  that  the  model  compare 
with  real  system  performance  parameters  within  ±3%.  This  figure  is  a  rule  of  thumb 
based  on  past  experience  and  judgment.  It  is  not  derived  from  any  analysis  of  a 
specific  problem.  (Comparisons  with  DIA  threat  baseline  data  are  made  at  the 
equivalent  of  the  SMART  functional  element  level.) 

Their  model  development  is  done  following  the  spirit  of  MIL  STD  2167A.  However,  the 
formal  documentation  is  generally  not  prepared  since  the  development  is  done  by  gov¬ 
ernment  personnel  for  their  own  use.  In  some  cases  a  model  may  be  sent  to  FASTC 
who  will  perform  the  comparison  with  the  DIA  baseline  data.  In  these  cases  FASTC  will 
prepare  a  formal  Acquisition  Validation  Report.  This  report  is  considered  sufficient  vali¬ 
dation  for  the  model. 
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INTERVIEWEE:  Gerald  B.  Bennett  DATE:  4  February  1993 

ORGANIZATION:  Aeronautical  Systems  Command,  Studies  &  Analysis  Division 
(ASC/XR) 

ORGANIZATIONAL  RESPONSIBILITY:  Provides  analytical  support  to  ASC  to  assess 
weapons'  effectiveness,  estimate  costs,  and  perform  preliminary  design.  Use  a  variety 
of  models  including  vulnerability,  survivability,  cost,  and  campaign  models. 

INTERVIEW  SUMMARY:  When  performing  an  analysis,  the  first  step  the  analyst  takes 
is  the  selection  of  a  model.  This  selection  is  based  on  what  type  of  information  is 
needed,  how  this  information  is  to  be  used,  what  money  is  available  for  the  assessment, 
and  how  much  time  is  available.  A  particular  model  is  selected  by  the  technical  leader 
who  is  responsible  for  the  assessment  and  who  must  justify  the  selection  to  the  program 
manager  who  requested  the  assessment. 

XR  hasn't  performed  a  formal  accreditation  in  the  past.  They  do  perform  verification, 
validation  and  sensitivity  analyses.  Within  XR,  each  group  is  responsible  for  a  selected 
set  of  models.  They  maintain  familiarity  with  these  models,  do  the  necessary  V&V  to 
maintain  confidence  in  them,  and  perform  analyses  using  them.  The  amount  of  V&V 
done  is  based  on  a  subjective  judgment  of  the  technical  analysts  who  are  familiar  with 
the  models.  They  try  to  perform  V&V  as  necessary  to  establish  a  consensus  within  the 
technical  group  that  the  model  results  in  a  particular  application  will  be  reliable. 

The  analysts  do  perform  a  lot  of  what  is  termed  “verification.”  They  select  simple  prob¬ 
lems  for  which  solutions  can  be  calculated  manually.  These  results  are  compared  with 
the  predictions  from  selected  model  components. 

Even  if  a  model  were  developed  in  accordance  with  MIL-STD  2167A  and  other  required 
standards,  ASC  would  still  want  to  do  their  own  verification  of  a  model  since  they  almost 
always  make  changes  to  a  model  prior  to  analytical  use. 

When  making  any  changes  to  a  model,  these  changes  are  documented  with  the  liberal 
use  of  comments  in  the  code.  They  insert  comments  at  the  beginning  of  each  subrou¬ 
tine  that  list  the  date  of  the  most  recent  change  and  describe  the  function  and  limits  of 
that  subroutine.  Updated  models  with  the  comments  inserted  are  fed  back  to 
SURVIAC. 

Another  problem  which  is  equally  as  important  to  ASC  as  model  validation  and  accredi¬ 
tation,  is  the  accreditation  of  the  data  bases  used  to  provide  input  data  to  the  models. 
The  information  contained  in  these  data  bases  is  generally  either  intelligence  informa¬ 
tion  or  data  related  to  friendly  systems.  These  must  also  be  accredited.  A  person  to 
interview  concerning  data  base  maintenance  is  Kevin  McCardle  at  Eglin  Code  ENY.  He 
maintains  an  Attrition  Data  Base  and  might  be  helpful  in  defining  the  SMART  data  base 
requirements.  Other  contacts  are  Frank  Campanile,  Larry  Boyd,  and  Larry  Beasely. 
They  can  be  contacted  through  Mr.  Weisenbach  at  the  same  location. 
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Another  source  of  information  on  accreditation  might  be  the  Avionics  Laboratory  at 
Wright  Laboratories  in  Dayton,  Ohio.  They  do  ECM  modeling.  The  J-MASS  program 
should  be  contacted  to  determine  what  their  plans  for  VV&A  might  be. 

Mr.  Bennett  did  not  know  of  any  instance  when  the  results  of  any  of  their  analyses  were 
rejected  because  the  credibility  of  the  model  was  in  question. 
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INTERVIEWEE:  Mr.  Michael  Weisenbach  DATE:  26  May  1993 

Dr.  Jerry  Arnett 

Mr.  Jordan  Wescott 

ORGANIZATION:  Aeronautical  Systems  Center,  Studies  &  Analysis 
Division  (ASC/XR) 

ORGANIZATIONAL  RESPONSIBILITY:  Provides  analytical  support  to  ASC  to  assess 
weapons'  effectiveness,  estimate  costs,  and  perform  preliminary  design.  Use  a  variety 
of  models  including  vulnerability,  survivability,  cost,  and  campaign  models. 

INTERVIEW  SUMMARY:  The  Studies  and  Analysis  Division  may  perform  studies  to 
support  either  ASC  or  the  System  Project  Officers  (SPOs)  directly.  They  may  also  per¬ 
form  studies  for  the  Air  Staff  or  the  Air  Combat  Command  (ACC)  since  the  ACC  is  re¬ 
sponsible  for  a  project  through  Milestone  1 . 

The  studies  assigned  to  XR  are  of  various  sizes  and  levels  of  complexity.  Depending 
on  size,  one  person  to  a  team  may  be  assigned  to  perform  the  study.  For  short  term 
studies,  which  tend  to  be  the  simpler  studies,  they  normally  use  the  "informal  process”. 
For  longer  term  studies  they  usually  follow  the  “semi-formal  process"  Whichever  pro¬ 
cess  they  follow,  they  have  an  agreement  with  the  study  sponsor  as  to  what  level  of  in¬ 
volvement  the  sponsor  will  have.  The  choice  of  process  depends  a  great  deal  on  the 
time  available  for  the  study  and  the  manpower  available  to  perform  the  work. 

For  most  studies  they  use  legacy  models,  the  primary  one  being  ESAMS.  They  feel 
that  legacy  models  are  valid  and  do  not  require  additional  V&V  unless  they  make 
changes  to  the  model.  If  changes  are  made,  they  do  testing  of  the  code  as  the  changes 
are  made.  If  these  changes  are  not  to  be  exported  to  other  users,  no  formal  V&V  is 
done  on  the  changes.  They  don't  document  any  of  the  testing  they  do  but  they  do 
document  the  changes  themselves  with  lab  notes.  Although  there  is  no  official 
requirement  to  distribute  information  on  their  in  house  changes,  MAJ  Behymer  is 
interested  and  is  sent  copies  of  any  ESAMS  changes. 

They  have  a  working  relationship  with  MSIC  who  have  done  a  comparison  of  select  Low 
Observable  (LO)  cases  between  IMARS  and  ESAMS.  The  results  of  these  compar¬ 
isons  have  not  been  documented  and  shared.  Flowever,  the  point  of  contact  for  further 
information  is  Klaus  Bletzinger  255-701  6. 

In  this  case  as  well  as  for  mode  comparisons  in  general,  the  results  of  such  compar¬ 
isons  may  not  be  available  since  the  comparisons  also  involve  system  comparisons. 
Such  comparisons  may  not  be  politically  tenable. 

They  have  developed  a  matrix  of  SAM  simulations.  This  matrix  includes  a  list  of  phe¬ 
nomenology  which  might  be  useful  in  structuring  the  data  base.  They  will  send  us  a 
copy  of  this  matrix. 
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The  interviewees  felt  strongly  that  a  validated  model  did  not  ensure  a  good  analysis. 
Good  data  is  needed  and  the  analyst  must  understand  how  the  model  works,  how  data 
is  treated  in  the  model,  and  how  the  model  outputs  are  generated.  In  this  way  the  ana¬ 
lyst  will  know  if  the  results  are  valid  for  the  particular  problem  which  is  being  analyzed. 

They  suggested  that  some  additional  information  might  be  gained  by  interviewing  their 
counterparts  in  the  Electronics  Systems  Center  at  Hanscomb  AFB.  They  suggested 
COL  O'Prey  ESC/XRP  as  a  point  of  contact.  They  also  suggested  that  LTCOL. 
Hathaway,  USAF/XOM  also  be  contacted. 
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INTERVIEWEE:  Christopher  Pfledderer  DATE:  1  October  1993 

Douglas  McCown 

ORGANIZATION:  Aeronautical  Systems  Command,  Air-to-Air  Systems  Analysis 
(ASC/XREWA) 

ORGANIZATIONAL  RESPONSIBILITY:  Provides  analytical  support  to  ASC  in  perform¬ 
ing  effectiveness  analyses,  end  game  simulations,  and  air  target  vulnerability  modeling. 
They  also  operate  a  MIL-AASPEM  modeling  facility. 

INTERVIEW  SUMMARY:  ASC/XREWA  does  not  have  any  formal,  published  accredita¬ 
tion  requirements  or  guidelines.  The  feeling  of  the  personnel  within  their  section  and 
their  own  feeling  is  that  any  model  in  SURVIAC  has  some  degree  of  acceptability.  The 
selection  and  use  of  a  model  for  a  particular  application  depends  somewhat  on  its  us¬ 
age  history.  Previous  usage  tends  to  build  confidence  in  the  model’s  fidelity  and  accu¬ 
racy.  After  listening  to  the  accreditation  requirements  briefing,  they  agreed  with  Messrs. 
Bennett  and  Weisenbach  that  ASC  will  use  either  process  1  or  process  2  in  performing 
an  analytical  study. 

Time  constraints  on  a  study  or  analysis  is  often  a  factor  forcing  the  selection  and  use  of 
an  available  model  for  a  given  study,  even  if  the  available  model  does  not  fit  all  of  the 
study  requirements.  The  requirement  for  some  answer  outweighs  the  need  for  accurate 
representation  of  all  the  conditions  surrounding  the  problem.  This  time  constraint  is  of¬ 
ten  dictated  or  passed  on  by  the  person  who  normally  would  accredit  a  model  (e.g.  the 
program  manager). 

In  their  opinion,  the  definition  of  acceptance  criteria  for  the  use  of  a  model  in  a  particular 
application  often  cannot  be  done  completely  prior  to  initiating  the  study.  It  often  may  be 
necessary  to  actually  begin  the  study  and  obtain  initial  results.  In  evaluating  these  re¬ 
sults  some  additional  criteria  might  be  uncovered. 

They  also  mentioned  that  some  models  might  not  be  totally  acceptable  (i.e.  they  might 
not  meet  all  of  the  acceptance  criteria).  However  the  model  might  still  be  used  since  it 
is  the  best  or  only  one  available. 

In  discussing  the  notional  process  which  is  described  in  the  briefing,  they  questioned 
whether  DOD  would  accept  accreditation  decisions  which  were  made  within  the  ser¬ 
vices  or  if  DOD  would  require  some  separate  justification  for  the  selection  and  use  of  a 
particular  model.  Their  experience  is  that  DOD  decision  makers  in  many  cases  will  not 
accept  the  justification  provided  for  some  study  results  which  include  model  usage. 
DOD  will  often  require  additional  sensitivity  analyses  to  prove  model  outputs  and  study 
conclusions  are  valid. 

ASC  often  used  historical  data  on  model  usage  as  a  basis  for  a  “de  facto”  accreditation. 
One  of  their  thrusts  is  to  document  any  models  which  they  use  that  don’t  have  users 
and  analysts  manuals. 


C-31 


ARS  INTERVIEW  REPORT 


INTERVIEWEE:  Mr.  Michael  Murray  DATE:  26  May  1993 

ORGANIZATION:  Wright  Laboratory  -  Electronics  laboratory 

ORGANIZATIONAL  RESPONSIBILITY:  The  division  develops  models  to  support  the 
laboratory  or  project  managers. 

INTERVIEW  SUMMARY:  Mr.  Murray  has  directed  the  development  of  a  model  called 
MOSAIC  which  stands  for  "Modeling  System  for  ASTE  Investigation  of 
Countermeasures".  ASTE  stands  for  Advanced  Strategic  and  Tactical  Expendables 
and  is  a  project  which  has  completed  DEMVAL  and  is  in  engineering  development.  He 
has  contracted  for  the  application  of  the  J-MASS  VV&A  process  to  the  VV&A  of 
MOSAIC.  The  task  was  started  after  the  model  coding  was  well  underway.  The  delay 
in  starting  the  VV&A  was  due  to  a  lack  of  funds. 

The  VV&A  effort  was  impacted  by  the  lack  of  timely  test  data  and  the  time  and  effort  re¬ 
quired  to  train  the  VV&A  contractor  personnel.  Although  they  were  involved  in  the  test 
planning  phase  and  agreements  to  share  data  were  worked  out,  when  test  program  dif¬ 
ficulties  occurred,  the  VV&A  test  objectives  were  the  first  to  be  impacted.  He  is  still  pur¬ 
suing  the  V&V  effort.  We  agreed  to  review  the  draft  VV&A  document  to  determine  what 
elements  are  similar  to  other  accreditation  requirements.  We  will  also  provide  com¬ 
ments  on  the  documents. 

MOSAIC  is  becoming  more  popular.  The  A/FX  program,  which  is  now  a  joint  program, 
has  requested  the  model  for  their  analysis.  The  ESC  Combat  Systems  Research  Lab 
will  have  the  responsibility  for  CM  on  the  model.  If  the  SMART  CM  study  personnel 
want  to  contact  some  of  these  CM  people,  Mr.  Murray  can  provide  a  point  of  contact. 

The  data  which  is  being  gathered  to  validate  MOSAIC  can  be  made  available  to 
SMART.  Since  it  is  IR  sensor  data  for  validation  of  the  IR  portions  of  MOSAIC,  it  should 
be  useful  in  validating  the  IR  portions  of  ESAMS. 
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INTERVIEWEE:  Mr.  James  Vernon  DATE:  8  July  1 993 

Dr.  Al  Murashige  (AFSAA) 

ORGANIZATION:  Headquarters  USAF/XOM 

ORGANIZATIONAL  RESPONSIBILITY:  This  staff  organization  has  been  established  to 
be  the  focal  point  for  modeling  simulation  and  analysis. 

INTERVIEW  SUMMARY:  Mr.  Vernon  indicated  that  the  XOM  office  was  recently  estab¬ 
lished  and  they  were  just  starting  to  deal  with  the  issue  of  developing  an  Air  Force 
VV&A  policy.  (He  is  responsible  for  M&S  VV&A  within  the  office.)  He  feels  that  drafting 
a  policy  would  be  very  simple  in  view  of  the  huge  amount  of  VV&A  policy,  guidance,  and 
philosophy  documents  which  already  exist.  He  indicated  that  implementing  the  policy 
would  be  the  difficult  aspect. 

His  approach  would  be  to  promulgate  a  trial  policy  and  attempt  to  implement  it  for  a  se¬ 
lect  set  of  models.  After  listening  to  a  presentation  on  the  interim  results  of  this  accredi¬ 
tation  study,  Mr.  Vernon  indicated  that  a  set  of  "decision  rules"  are  the  key  to  a  success¬ 
ful  accreditation.  His  meaning  for  the  term  "decision  rules"  is  a  set  of  guidelines  which 
should  be  followed  to  allow  a  decision  maker  to  determine  if  a  M/S  was  suitable  for  a 
particular  application.  These  decision  rules  are  essentially  similar  to  a  checklist  of 
things  to  look  for  in  a  M/S.  It  also  embodies  the  concept  of  the  acceptance  criteria 
which  have  been  identified  by  several  model  users  as  being  a  critical  factor  in  model 
VV&A. 

Mr.  Vernon  provided  a  brief  description  of  the  XOM  mission  and  functions  as  well  as  an 
office  organization. 
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INTERVIEWEE:  Mr.  Kevin  Crosthwaite  DATE:  27  May  1993 

Mr.  Dennis  Detamore 

ORGANIZATION:  Booz-Allen  Hamilton  (SURVIAC) 

ORGANIZATIONAL  RESPONSIBILITY:  SURVIAC  has  the  responsibility  for  maintain¬ 
ing  a  library  of  approved  survivability  models.  Booz-Allen  Hamilton  (BAH)  operates  the 
library  under  the  direction  of  a  SURVIAC  steering  committee. 

INTERVIEW  SUMMARY:  When  a  model  is  recommended  for  general  distribution  by  a 
cognizant  authority,  the  SURVIAC  steering  committee  decides  if  it  should  be  included  in 
the  library.  When  it  is  approved  by  the  steering  committee,  BAH  takes  a  copy  of  the 
model  and  runs  it  on  their  computer.  They  run  several  test  cases  for  which  they  have 
results  from  running  previous  versions  of  that  model  or  similar  models.  (These  test 
cases  are  provided  by  the  model  developer.)  If  the  results  are  consistent  with  previous 
runs  the  model  is  made  available  for  distribution  to  users. 

They  pointed  out  that  the  copy  of  the  FEAR  which  they  received  and  reviewed  did  not 
contain  the  type  of  information  which  they  expected  and  which  users  need  to  ensure 
that  the  model  is  operating  correctly.  Specific  comments  on  the  draft  FEAR  were  pro¬ 
vided. 
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INTERVIEWEE:  Donald  Martinovich  DATE:  27  May  1993 

Dan  Sammons  (SVERDRUP) 

ORGANIZATION:  SIMVAL  Program  and  J-MASS 

INTERVIEW  SUMMARY:  The  SIMVAL  program  validates  simulators  against  DIA  base¬ 
lines.  They  obtain  their  data  from  the  CROSSBOW  Executive  Committee.  If  sufficient 
data  is  not  available,  they  develop  the  data  and  coordinate  with  the  lead  service 
Science  and  Technology  (S&T)  to  gain  approval  or  concurrence  on  the  data. 

Their  comparisons  between  the  simulator  and  the  DIA  baseline  is  done  at  four  different 
levels.  They  compare  1)  descriptions  of  the  systems,  2)  functions,  3)  parameters  de¬ 
rived  from  specific  model  runs,  and  4)  performance  factors  such  as  frequency  response 
or  step  response.  Recently  they  are  leaning  toward  performance  comparisons  to  vali¬ 
date  a  simulator.  If  the  comparison  results  are  not  satisfactory,  they  proceed  to  other 
levels  of  comparison  to  find  the  problem  in  the  simulator.  Currently,  they  are  doing  a 
comparison  between  ESAMS  and  IMARS  which  is  the  DIA  approved  baseline. 

If  there  is  no  DIA  approved  baseline  and  they  cannot  develop  the  necessary  data,  they 
have  six  degree  of  freedom  (DOF)  models  which  they  use  for  generating  data  to  com¬ 
pare  with  simulator  outputs.  This  data  is  not  DIA  approved  and  they  only  use  these  6- 
DOF  models  if  no  other  data  is  available. 

Their  V&V  process  includes  efforts  by  both  the  developer  and  an  independent  group. 
They  prepare  a  test  matrix  which  considers  both  observed  performance  of  the  model 
and  model  requirements.  The  testing  which  is  done  is  similar  to  module  testing  which 
we  do  as  part  of  verification. 

They  mentioned  that  they  use  the  standard  validation  criteria  which  is  found  in  the  ap¬ 
pendices  of  the  Threat  Simulation  Program  Plan  which  is  available  from  CROSSBOW. 
A  quick  review  of  these  appendices  shows  that  these  "criteria"  are  lists  of  test  parame¬ 
ters  and  their  relationships  to  each  other.  The  "criteria"  do  not  include  fidelity  require¬ 
ments  for  any  of  the  parameters.  The  information  in  these  appendices  would  provide 
little  insight  into  what  the  fidelity  requirements  might  be  for  different  classes  of  applica¬ 
tions. 

The  authority  and  guidelines  for  performing  their  V&V  are  spelled  out  in  DODINST 
5000.2,  AFR  8025  and  DOD  5000.3-M-6.  This  latter  document  which  is  titled  "Threat 
Simulation  Program  Policy  and  Procedures"  is  now  canceled  but  they  use  it  to  guide 
their  efforts.  The  Navy  persons  who  are  involved  in  SIMVAL  is  Jim  Fusner  at  China 
Lake.  His  number  is  939-2878.  Al  Dunkerly  of  ARI  who  provides  support  to 
CROSSBOW  is  another  person  who  might  be  a  worthwhile  point  of  contact.  His 
number  is  (703)  243-8448. 
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INTERVIEWEE:  Mr.  John  Freeman  DATE:  9  June  1993 

ORGANIZATION:  Defense  Modeling  and  Simulation  Office 

ORGANIZATIONAL  RESPONSIBILITY:  Acts  as  a  full  time  focal  point  for  information 
concerning  M&S  activities.  Responsible  for  promulgating  USD(A)  directed  policy,  initia¬ 
tives,  and  guidance  to  promote  cooperation  among  DOD  components  to  maximize  M&S 
efficiency  and  effectiveness. 

INTERVIEW  SUMMARY:  A  meeting  was  held  with  Mr.  John  Freeman  in  the  DMSO. 
The  purpose  of  the  meeting  was  to  determine  what,  if  any,  information  concerning  ac¬ 
creditation  requirements  could  be  obtained  from  that  office.  Mr.  Freeman  opined  that 
the  services  have  been  responding  to  DOD  and  are  engaged  in  developing  policies 
and  procedures  addressing  M&S  accreditation.  Now  the  services  are,  in  effect,  asking 
DOD  for  an  opinion  as  to  whether  the  policies  and  procedures  which  have  been  devel¬ 
oped  meet  DOD  expectations.  In  response  to  this  question,  DOD  is  now  wrestling  with 
the  issue  and  is  preparing  a  directive  or  instruction  which  expresses  the  DOD  position 
and  viewpoint.  Currently,  the  DMSO  does  not  have  any  expressed  policy  or  guidelines 
on  either  the  process  to  be  used  or  the  information  required  to  support  accreditation. 

Mr.  Freeman  explained  that  the  DOD  organization  is  undergoing  change.  A  new 
organizational  structure  has  been  drafted  and  is  being  reviewed.  Since  the  DMSO  is 
not  shown  on  that  new  diagram,  he  is  unsure  as  to  the  future  of  the  office.  However, 
CAPT  McClure,  the  present  OPNAV  812,  has  been  named  as  the  new  director  of 
DMSO.  Therefore,  they  expect  to  continue  in  their  present  role,  albeit  in  a  new  organi¬ 
zational  structure.  (As  a  side  note,  CAPT  Tom  Travis  will  be  replacing  CAPT  McClure 
in  N-812.) 

Mr.  Freeman  also  provided  some  background  information  on  the  DMSO  and  how  it 
came  into  being.  He  pointed  out  that  the  director  has  discretionary  control  over  about 
20%  of  the  DMSO  funds.  There  is  a  formal  process  for  submission  of  proposals  to 
obtain  some  of  this  discretionary  funding. 
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